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ment were discussed. Alternative techniques for computer 
evaluation and for workload description were contrasted. 
Proposed solicitation methods, computer procurement plans 
and computer performance measurement and evaluation techniques 


were surveyed. 
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Ein the past twenty years, the computer industry has 
sustained a technical reevaluation unrivaled in modern 


history. The computer has become the greatest management 


DC 


ROO! of our time 2 Yet its proper application contains many 
alls, as case after case of dramatic failures testify. 
One of these dangers is improper equipment selection. When 
managers thoughtlessly procure equipment as a natural process 
item, they can easily preclude any possibility of success 
simply by buying the wrong equipment. 

The computer selection and evaluation process has become 
a complex one, requiring detailed attention; it can involve 
hundreds of technical as well as nontechnical considerations. 
Today's computer systems are typically very complex and 
extensive in composition and operation. Academic and admin- 
istrative users of computer systems have traditionally left 
most of the considerations in systems selection to technical 
personnel. AS a result, many user needs have gone unsatis- 
fied. Technicians have become frustrated because they often 
found out too late, if at all, wnat user needs and priorities 
really were. In addition, many technicians have had diffi- 
culty in communicating to relatively untrained users a 
complete understanding of the actual capabilities of various 
computer systems. 

Tne remarkable technical reevaluation in the industry has 


led to the creation and ultimate availability of a large 
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number of unique computer systems. When one considers She 
vast number of peripheral devices available with these basic 
systems, added to the various special purpose and analog 
devices, the number of unique computer configurations is 
almost infinite. Hro this confusing marketplace the prospec- 
tive buyer brings his enthusiasm, but not a disciplined 
Approach to the selection process, Technical progress and 
new application opportunities occur so fast that any organ- 
ization's equipment strategy should come under frequent 
review. There are any number of circumstances that dictate 
the requirement for an open evaluation of computer requirements, 

The competitive nature of the computer industry has caused 
vast technical changes during the last decade; during this 
period, the industry has moved from punched card orientation 

to online communications. Relatively fast memories and large 

capacity direct access devices with relatively fast access 
times and sophisticated operating systems, also have been 
made available with present-generation data processing sys- 
tems. Large funds, placed in research and development, have 
led to ever increasing numbers of new systems, each one 
larger in capacity, faster, more capable, with more software 
than the last ones. The industry is dynamic. The pressure 
on the present user to move from his presentiy obsolete sys- 
tem to the newer, more powerful system is logical and 
demanding of analysis. 

Allied to the technical changes in the data processing 
field are economic changes. Equipment is now being made 


available at substantial cost reductions. One can procure 
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a third-generation system at 40 to 60 per cent of the cost 
of equivalent older equipment. The user who formerly made 
changes only when present equipment was incapable of satis- 
fying processing needs is now forced to compare the new 
added equipments on the basis of technical and economic 
advantages. The continual changes in the field require such 
an analysis on a periodic basis [Ta tham 1969, Yearsley 1973, 
Thrussell 1976, Joslin 1977]. 

| The purchase of a computer system is a major considera- 
tion for any company, but it is often approached and inves- 
tigated in a superficial way. Computer salesmen are called 
in, but in many cases the company does not get maximum usage 
from a computer purchased from a high pressure salesman. 
Disillusionment spreads among the users, and it is quite 
often based on preconceived notions of the purchased computer. 
It is considered a "save all and do all" type deal. You can 
save all this money because the computer will do all the 
tedious work. They fail to realize that even though a 
computer can produce some material 24 hours a day, it does 
not mean that the computer is alive. Rather, the computer 
is dead simply because management is dead to possibilities. 
The manager should be completely aware of the computer's 
tential, Since he finds himself sitting in on or leading 
many management teams. He should be aware of the intangible 
problems that may cause the computer to become economically 
unfeasible, or which may creep in later if the computer is 


implemented. 


N O 


| 


The effectiveness and potential of any computer-based 
system is strongly influenced by the design of that system 
as well as the choice of equipment. lhus the scope of the 
equipment available must be evaluated when selecting the 
equipment in order to understand how a particular choice 
affects the planned system. 

The organization entering the computer world for the 
first time normally lacks in-house talent since it may not 
have any people having background in computer applications. 
This leads to major and often complete reliance on the 
marketing wiles and brochuremanship of the computer manu- 
facturer. Like the uninformed buyer in most fields, the 
uninformed Automatic Data Processing (ADP) user normally 
contracts with the reputable firm, hoping that the firm is 
so interested in the user's unique applications that he will 
get unusual service. Thus, a major decision, capable of 
EM ecting the future competitive position of the firm, is 
quite easily turned over to the equipment manufacturer. A 
much better solution can be to seek outside consulting helo. 
Even a substantial investment in consulting fees can pale in 
Mea tion to the cost of a poor selection, 

The selection of a computer and manufacturer is one of 
the major decisions to be made in formulating an organiza- 
tlon's computer policy. Also for the following reasons the 
decision is important; the equipment itself is probably the 


largest individual expense in the computer department, and 
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the selection of manufacturer and a specific computer sets 
a constraint on system development that will last at least 
B. years [Clifton 1969, Tatham 1969, Wooldridge 1973, 
Procop 1976, Withington 1974, Sabol 1972]. 
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۱۳۳ ۷۰۲۶۰۱۷۰ ۱۷۱۷۰۱۱۸۰۰۰۰٠٦۲٣ TO SELECTION 


Once it is clear that evaluation of computer needs is 
necessary under a particular set of circumstances, a series 
of studies as preliminary steps to the actual process of 
selection should be initiated. These steps are time- 
Ponsuming, but they are an essential tool that managers can 
use to make an accurate evaluation of the equipment require- 
ments. This study is designed to determine the cost effec- 
tiveness of various solutions to ihe data processing problem, 
These alternative solutions will include the present system, 
plus various combinations of data processing systems. For 
each alternative, there will be a distinct systems approach, 
as well as an appropriate cost, 
| ۲٣۰٠۰٦٠٠٢ being too ambitious and without being too 
restrictive, the planning for a data processing system must 
look at today but it must also assess tomorrow. Some of 
these tomorrow's requirements can be reasonably forecast 
today. Many of them result from growth patterns of the 
enterprise. Such growth patterns and changes are fundamental 
in nature and demand a coordinated management information 
System which can: screen all company data and prepare 
reports according to requirements, provide faster factual 
information in a dependable and management oriented manner, 
help management with both present and future information 


needs and provide data or events before they happen. 
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It is important to distinguish between what management 
wants and what it needs, The comparative analysis will allow 
management to exercise its judgment as to the feasibility of 
new system development. It is important to remember that 
this new system cannot, and should not, be undertaken on the 
basis of cost savings alone. The value of information, 

መ ንክ difficult to quantify, must be considered in the 
study. The incremental cost between two alternatives will 
often be more than offset by the value that the increased 
information will give to the decision-maker. It is also 
important that these factors be weighed at high levels of 
the organization's management. 

“Once an approach has been accepted by management, a 
macrosystem design effort begins» Information flows are 
defined, inputs and outputs are determined, and files are 
developed. 0 all the basic data required by the 
system are defined and gross flow charts developed, the 
system desisner is limited, at this point, by his lack of 
knowledge of the specific hardware configuration. However, 
it is during the applications study that the model vrhich 
will ultimately become the operational system is created, 
Meeufficient effort at this point can only lead to poor 
equipment selection and development of a stunted system. 

Now, failure to undergo the detailed information study can 
only mean postponement to the time when maximum effort must 


be allotted to software, procedures, and training development. 
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Ihe importance of the application study cannot be over- 
emphasized. <The results of the study are normally appended 
to the system specifications for the hardware manufacturers 
to use as a basis for proposals. Because of the importance 
in defining the model system for vendor proposals, it is at 
the applications study level that professional data process- 
ing support must be made available. Decisicns on offline 
versus online systems, disc versus tape, two-channel versus 
we channel, basic processor speed, and memory size, 
require highly skilled systems analysis personnel thoroushly 
familiar with both the state cf the art and the function to 
be automated. Unfortunately, such zersonnel, almost without 
exception, are in very short supply [Clifton 1969, Joslin 
2077, Tatham 1969, Chora fas 1967]. 

In addition, the complexity of modern systems is so 
great that it is almost impossible for a systems analysis 
team to consider all the major alternatives. Six years 
computer experience with a company may be an excellent base 
for systems analysts or data processing management but again 
unfortunately it seldom covers experience in the process of 
computer selection or in the scale and technology about to 
be investigated. 

The best solution to the above problem is the use of 
Ms tion techniques to assist in the system design. With 
this technique, a description of the user's workload (files 
and programs), along with the appropriate hardware and 
software characteristics of the configuration to be simulated, 


45 used as input to a sophisticated computer program. The 
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program is a generalized mathematical model of computer 
processing, which enables the workload to be simulated and 
valuable performance data to be collected. In this way, the 
system design analyst can try out different ideas and 

EN roaches in order to arrive at a good system design. 

Use of the simulation approach can drastically reduce 
the amount of elapsed time required for the applications 
study effort, and results in a better system design by pro- 
viding more analysis than is possible with a manual method. 
If this approach is taken, caution should be exercised to 
ensure that the simulation model accurately handles the 
essentials of third-generation processing. In order to be 
most useful for system design, the simulator should allow 
easy man/machine interaction by: fast turnaround time, out- 
put results oriented toward suggesting system improvements, 
and flexibility allowing design or configuration changes to 
be made easily [Joslin 1977, Graham 1973, Clifton 1969 
Martin 1973]. 


o 
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RR CECIEICATION DEVELOPMENT 


Mie ን ol specifications plays an important role in 
system evaluation. Development of specifications, which will 
be released to all interested manufacturers, is a crucial 
part of the process. Specifications for a computer system 
can be prepared in a number of different ways.‘ The specifi- 
cations should be general enoush to assure wide competition, 
yet specific enough to delineate the user's requirements 
clearly.» The mad resulto cae application study is a 
documented model system. This system should become cart of, 
and treated by, the specifications as a point of devarture 
ሺን ት» ከበ ከ ከ1ር ከ each manufacturer is free to use his own ingenuity 
and brain power to develop a superior system, oriented to 
own equipment. Proper control of this process is main- 
tained by establishing the constraints witain which each 
manufacturer must work. 

The design of the system specifications should be the 
starting place for developing any computer selection plan. 
It should define what is sought in the way of a computer 
system, give the system requirements for the various appli- 
cations, and give a detailed description of each step of 
each application. The system specifications reflect the 
findings of the system study team. The final choice can be 
detrimental to both the company and the vendors if proper 


care is not taken in the preparation o? the specifications. 
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The specifications must reflect the actual applications to 
be handled by the system, and must not contain poorly 
mit out limitations whlch are to be imposed on the 
Een. Also they should not be directed too much toward a 
specific systems approach [Joslin mov. Chora fase 957, 
Wooldridge 1973]. The several methods are described in the 


Ml owing sections., 


A. GENERAL SPECIFICATIONS 

General specifications are nothing more than the findings 
of the analysis, a description of the jobs to be done: the 
inputs, the desired outputs, and any other pertinent param- 
eters. (As an example of cutputs, consider a stock level 
reporting. Approximately 7000 items are sold daily; for 
every item sold, a description card is produced, an excep- 
tion listing of all items below a specific amount is to be 
created daily, and a total listing of all types and amounts 
of inventory items is to be prepared monthly.) 

The general specifications give each vendor a chance to 
build a system which makes optimum use of the features of 
his system. Each vendor is free to use to the utmost any 
experience and ability he has to prepare the proposal. 
General specifications thus make maximum use of the vendor's 
system analysts and permit nim complete freedom to produce 
the best possible system for the user. Rather than relying 
on the limited experience of the company's two or three 
analysts, the problems are tackled by the vendor's top 


analysts, who are more adequately geared for such work. 
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Innovations to the system may well be the result of such an 
exercise. Or, a smaller, less expensive system than was 
monent possible might be proposed as a result of some 
exceptionally good system work by a vendor. The vendor may 
propose that some of his package application programs could 
Setisty many of the system requirements. With software 
costs equaling hardware costs on today'S systems, the result- 
ant cost savings could be important to both vendor and user. 
From a systems viewpoint, the user has much to gain by 
relying on general specifications io describe the applica- 
tions and needs. However, at the same time many problems 
are created. With a set of general specifications, a user 
should expect to spend many hours with vendors and their 
representatives, discussing possible systems approaches. 
Countless hours will also be spent trying to verify the 
systems proposed by the vendors in order to ensure that they 
will be able to handle the required applications. It is 
also important that the system concepts pronosed are thor- 
oughly understood by the user. Whichever system is selected. 
the vendor's representative will not be delivered with the 
system. It will be the user's responsibility to turn the 
۱ الا‎ into reality. 
General specifications also prove awkward and difficult 
as standards by which to compare competitive proposals and 
to select a winner. With general specifications, system 
rewards can be great,in terms of improvements to the computer 
System, but the difficulties of evaluation can also be 


great [Yearsley ከእ ی‎ 1967, Tatham 1969, Joslin iom 
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B. DETAILED SPECIFICATIONS 

Detailed or specific, specifications are just what their 
meme implies; each and every step to be taken in each of the 
@eplications is spelled out. Usually, the synthesis that 
was used in developing the flow charts for cost determina- 
Rn during the system study is repeated step by step in the 
۳۰٠۰۰-۰۰٣٦٦673. Detailed specifications must be written 
Mery carefully to ensure that they do not become machine- 
oriented rather than application-oriented. Machine-oriented 
wec fications might discriminate against some vendors and 
thus unintentionally deprive the company of the best system. 
Since detailed specifications require the vendors to con- 
figure their systems exactly as specified, the systems design 
work for the vendors is simplified, but allows them little 
freedom to fit the applications to their computers. The 
computers must be fitted to the applications. 

Since detailed specifications are completely descriptive 
and are fully and uniformly defined to all vendors, they 
have a definite advantage for the user. Thus, there should 
be little time wasted in talking with vendors. No system 
will be proposed that is far inferior to the system repre- 
sented by the specifications. The submitted proposals may 
ve more easily compared, verified, and evaluated since the 
proposed Systems must all be identical, matching the steps 


set forth in the specifications. With detailed specifications, 
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the systems proposed can be no better than the system 
S@ectiied, although the trouble involved in obtaining it 


should be minimized. 


Ome OTHER APPROACHES 

Specifications do not have to be either general or 
detailed; they may be at any level in the general-detailed 
range, in which a certain amount of synthesis may be done 
by the user (and the user specifies this part in detail), 
and a certain amount may be left to the imagination of the 
vendor. Such a method may be called a modified-detailed 
ication. its relative merits اس‎ upon a rule of 
direct proportions: the more general the specifications, 
the greater the chances of obtaining a superior system; the 
greater the degree of detail in the specification, the easier 
the proposals will be to handle. The user can Set the level 
of modification of a full-detail specification by the degree 
of synthesis he gives to the vendor. 

The combination of general and detailed specifications 
ought to be used in preparing system specifications. In 
this method, the general specifications are given as the 
guidelines to be followed in preparing the proposal, and the 
detailed specifications are given as an example of how the 
applications might be handled. The use of the detailed 
specification as an example serves a threefold purpose: 

1. It clearly indicates the activities and functions to 
be performed in each of the applications, and answers many 


questions that the vendor might otherwise have to ask. 
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2, It becomes a common starting point for all vendors. 
The example may be modified differently by the contending 
vendors but they are all departing from the same basis. It 
۳۳۲ ۱۷۵8 29 indication of the level of sophistication 
being sought in the proposals. 

3. It gives the small vendor something good on which to 
base a bid without necessitating a full system analysis 
ort. 

Preparation of detailed specifications forces the user 
taco the kind of thinking that the vendors will have to 
engage in. Since the user will be doing the thinking first, 
he should discover any problem areas before the specifica- 
tions are released to vendors. This naturally tends to make 
for smoother relations between the vendors and the user. 
Proposals submitted in response to this combination specifi- 
cation should all present solutions as good as the detailed 
Specification approach [Tatham ከፕ ፕ ያታ = vearsiey 1973, Joslin 
en, Chorafas 1967]. 

Usually, the type of application to be handled by the 
computer system indicates the type of specifications to be 


used. 
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IV. LIMITATIONS IMPOSED ON SELECTION 


The limitations to be imposed on the computer system 
selected should have been uncovered in the system study. 
There are two kinds of limitations: mandatory and desirable. 
The distinction between these two categories should be that 
he items listed as mandatory requirements are those items 
that are essential to the implementation of the company's 


needs. 


A. MANDATORY REQUIREMENTS 

Mandatory requirements should be stipulated by the speci- 
ions, in Order to protect the user from considerations 
of proposals which will not satisfy basic needs. By defini- 
۳ 3 proposal will receive no consideration if it fails to 
meet any one of the mandatory requirements. Tre less strin- 
gent the mandatory requirements, the more likely it is that 
any given manufacturer will be able to compete in the 
procurement. Some examples of such mandatory requirements 


ከሚ shown in Figure 1. 


B. DESIRABLE VERSUS MANDATORY REQUIREMENTS 

The desirable features are only those items which would 
make the completion of the company's mission easier. Upon 
submission of a proposal by a vendor, and in consideration 
of the limitations of both categories designated by the user 


in his solicitation of proposals, the failure of the vendor's 
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EXAMPLES OF MANDATORY 
REQUIREMENTS 





Minimum turnaround time 

10 9 و ط‎ r processing time 

Compiler availability and requirements 
Operating system requirements 
Comp ده و‎ 7 

Expansion requirements 

Translation requirements 

Site constraints 

Hardware constraints 

ecarpment cualification 


Special software 


ER 





Droposal to possess some desirable feature as designed should 
invoke some penalty upon the proposal, although it would 
continue to be considered in the process of selecting the 
most advantageous proposal. 

Computer acquisitions have one thing in common: in 
most cases, the company wants the best system it can find 
for the lowest possible cost. In order to stay with the 
IS. possible cost, most users are interested in general- 
 ጵ ዎሮ computers. Special-purpose computers car be expected 
to run anywhere from 50% to 700% or more in cost than 
em pal-purpose ones [Rubin 1971] . ከ ከ ቱ፡ ተ ።ሪ።ሮን a 
seneral-purpose computer implies certain features that are 
not readily changeable. The general-purpose computer must 
be taken largely as it comes from the plant. Thus the user 
is put into a position analogous to one who. has spent years 
drawing up blueprints for a new house, but who suddenly 
finds himself witn an immediate need for a house. He can 
have the house built to his blueprints, which will prove 
I expensive due to unique building cost and the cost of 
temporary housing, or he can look for an existing house that 
fills his needs. If in searching for an already built house 
he is looking for one that exactly matches his blueprints, 
he may well have to go without a house. But if he is willing 
to not adhere strictly to the specifications, looks for 
houses with similar room layouts and other features, and 
settles upon the one most closely matching his blueprint, 
then he will have a house, which is his major requirement 


[Chora fas Doc Ooi 1977, Tatham 1969]. 
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C LIMITING CONDITIONS 
Now that the dangers inherent in mandatory requirements 
have been discussed, the several classes of limiting condi- 
Ems can be described. 
ዕ) Cost of the System 
A primary consideration in the study of any computer 
system is the amount of money that the company is willing to 
mena. Mandatory cost limitations may produce an effect 
۳--٠6٥ to the one desired. Costs should be reserved for 
use as evaluating factors and not as limiting factors. If 
uly mandatory condition exists, such as that a fund of 
so many dollars, with provision against its increase, has 
been set aside for procurement of equipment, then the condi- 
mDould be stated. Absolute funding of this sort is 
unlikely to exist. System costs normally should be handled 
BEM desirable limitation. 
2. Due Dates 
One عو‎ first things encountered in a system 
study will be the existence of due dates. As a matter of 
fact, these dates are not often really mandatory and they 
should not be treated as mandatory requirements but as 
desirable ones to express wishes and not commands. 
(3) Application Capabilities 
The purpose of the study is to get a system capable 
of handling the applications specified. However, the possi- 
Eh. remains that some of the desired capabilities are not 
avallable. It is better to handle application requirements 


as desirable limitations. 


ag 





4, Responsiveness 
Mass isa limitation factor akin to application 
requirements.  Responsiveness requirements should be regarded 
as desirable limitations, as are most application capabil- 
ities. Since responsiveness is the element of greatest 
interest to the top executives of the company, it can easily 
be made absolute by the statements of the top executives. 
ERO Compatibility 
The compatibility of the old system with the new 
should definitely be considered, but whether it should De 
mandatory is questionable. There is always temptation to 
make compatibility with the present system a mandatory 
requirement of the new system. 
6. Vendor Support 
Often seen in specifications, and stated as mandatory 
requirements, are conditions which relate to the tyve of 
support that must be available from the vendor whose proposal 
is accepted. They can be mandatory or desirable limitations 
murs on the particular cases of applications. 
۲۳۱ Reliability 
Reliability is a condition inserted into most speci- 
fications as a mandatory limitation, generally expressed in 
such terms as "the system must have 95% uptime”. Specifica- 
tion should state that reliability will be a factor in 
evaluation and selection of a proposal, however, the value 
of reliability should not be exaggerated or expressed in 


such absolute terms as to prevent the exercise of judgmert. 
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۴۰۰-۰٠1777 1s to be used as a mandatory requirement, then 
some means of measuring it will have to be determined.  Usu- 
۳۲۳ the problem is chat the measurement of the reliability 
would take more time than could be allowed. Or, as in the 
case of real time systems where reliability really isa 
Rca tory limitation, complete redundancy of the system may 
be required, 
8. Space Requirements 

Ihe dimensions of systems may change from selection 
to selection. Space requirements is a desirable limitation. 
In most cases the immediately available space could be 
extended or new space found. À proposal should not have to 
be discarded just because it requires more than the allotted 
space, especially when the allotment might have been deter- 
ከኔ ከ SN aroiltrarily or thoughtlessly. 

©. Input/Output Requirements 

Input/outout requirements are expressed in terms of 
forms or formats to be used, number of copies to be prepared, 
and the like. Input/output requirements which are limiting 
conditions should be restudied and reappraised. They should 
be viewed in two ways: 

a. What will happen to the system if these require- 
ments are changed? 

b. If these requirements remain unchanged, can 


vendors meet them? 


Sep 





ከ Other Limitations 
Many other limitations may appear ina specifica- 
Seen. Ine deciding factor in establishing them as mandatory 
requirements is whether eacn item is important enough to 
Wemreant throwing cut the proposal completely if it fails to 
meet the limitation to any degree. This happens rarely, but 
it can happen [Chorafas 1967, Tatham 1969, Joslin 1977, 


Wooldrige 1973]. 
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V. PROPOSAL EVALUATION TECHNIQUES 


A computer evaluation methodology is simply a planned 
method for selecting the most satisíactory comruter system 
from a number of satisfactory Computer systems, The evalua- 
tion methodology tries to assure that all of the computer 
mens in the final phase of selection are satisfactory; 
i.e., that tney meet all the basic requirements of the 
solicitation and that they are what the vendors represent 
tiem cto be. When thought of in this sense, evaluation may 
EUR lixe a rather trivial task, since any resultant 
ከየ Ton will, by definition, be satisfactory. However, 
there are different degrees of satisfaction and difíerent 
Bor people to be satisfied. 

In evaluation, higner levels of satisfaction and satis- 
faction of other than the basic requirements are considered. 
An important group that must be satisfied with the evaluation 
process is the vendors. The vendors will not spend the time 
and money necessary to bid the "satisfactory systems" that 
get evaluated, unless an evaluation methodology has the 
۶۱۳ 3 Of being fair and unbiased. If they have bid but 
feel they have not been fairly evaluated, they will protest 
Ene Selection. 

Within the government, or any large organization, a 
protest can lead to considerable embarrassment for the 


procuring activity if upheld and, in any event, will consume 
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NE St deal of time and effort in resolving the protest. 
Therefore, a selection methodology must be satisfactory to 
the vendors as well as to the procuring activity |Auerbacn 
1975, Clifton 1969] . 

The organization of the evaluation must be carefully 
structured so that the participants are aware or their 
individual areas of responsibility. A hierarchical arrange- 
ment is necessary in order to have increasing levels of 
responsibility as the decision areas become broader. 

In order to select the best computer system after speci- 
fications have been determined, the following need to te 
EsuEdered: 

* possible computer systems; 

* by whom the selection is to be made; 

* selection methods; 

* the criteria used in the selection process and their 
relative importance. 

Ihe information to be used in selection methods can be 
obtained from: 

* published surveys and reports, 
* service and product publicity material, 
* hardware, operating system and program documentation, 
* managerial, sales and technical staff, 
* in-house staff and other users of the machine or service 
[webster and Johnson Near 
To give an idea about competitive selection time, 


Succeeding steps in the competitive selection process for 
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ከሚ ከክ are Outlined in Table 1, which shows the estimated 
time to accomplish each step. The complexity of the system 
Beane acquired generally determines the length of time 
Mee@uired at each step [ Prokor 19764. 

An evaluation methodology should: 

* consider those items or features wanted but not manda- 
Eent. 

ever all the items or features desired, 

* facilitate the establishment of meaningful zrd under- 
standable relative values between all the desired items, 

* require the completion of the vrevious criteria before 
the solicitation document is completed, 

* permit disclosure of all desired items and their relative 
values to the vendors, 

* incorporate systems life costing. 

An evaluation methodology that satisfies: 

መሙ ስ ከ che listed criteria is a SUPERIOR methodology, 

* five of the listed criteria should be considerec a GOOD 
methodology; but before settling for it, a superior methodo- 
logy should be sought, 

* only three or four of the listed criteria may be consid- 
ered a FAIR methodology, but it should be possible to find a 
better methodology, 

* less than three of the listed criteria would have to be 
considered a PCOR evaluation methodology and should not be 


used [Auerbach 1975, House 1976, Chorafas 1967]. 
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TABLE 1 


۱۳۳۲ ان‎ ብህ CT ከህ TIME FRAME 





Draft request for proposals for 30-90 
ao proved project 
Release of draft for comments 30 
۲۲ ۱۵۳۲ of request for proposals 30 
Response to request for proposals 689 9 
Evaluation of proposals and benchmark 30-120 
Administrative time after evaluation 20-60 ፲ 
Installation of equipment after 90-270 


contract award 


Range: 290-720 Days or 10-24 Months 


Prokop 1976 
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A. TECHNIQUES FOR THE EVALUATION OF COMPUTER SYSTEMS 
There are several techniques for the evaluation of compu- 
systems. They generally fall into one of two categories. 
Either they are very simplistic in that they tend to ignore 
most of the criteria listed previously, or they are ینگ ےگ‎ 
Mnd incorporate most of those criteria [Auerbach 1975]. 
1. Simple Techniques 
Simplistic methodologies are better xnown but less 
Exesstul techniques. Theoretically, they are not worth 
much discussion, but they illustrate the neec for the more 
sophisticated techniques. 
a. sole Source 

"I have been happy with this vendor. Why should 
I change?" It is possible that one might be happier, for 
less money, with another vendor. 

b. Subjective Judgment (Overall Impression) 

Probably the most frequently used evaluation 
approach is to have no preestablished epproach, just some 
general statement such as: "When the proposals come in, an 
unbiased group of evaluators will look through them and pick 
the one that provides the most benefits at the lowest prices." 
People who advocate this approach to computer selection will 
ridicule any attempt by a prospective user to preestablish 
which items he must consider in an evaluation before he 
receives the proposals to be evaluated. They will ridicule 
the prospective user even more if he attempts to establish 


specific values for each of these itens. 
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Prospective evaluators will argue that until the 
prospective user has received all the proposals for computer 
systems, he will not know all the items. They will point 
out, for example, that if all the vendors propose any given 
EN "tem, then its importance to the selection process is 
negligible; whereas even a minor item can have significant 
O nce on the final decision if it is proposed by one or 
two of the vendors but not by all, These prospective evalu- 
E want to make their selection first and then justify 
their evaluation. 

This procedure is a comfortable one for the 
evaluator since he is not forced into doing any advanced 
planning. Almost any vendor who meets the mandatory require- 
ments could be selected as the winner under those circum- 
stances. All it takes is an evaluator who is clever with ۱ 
words and who can accentuate the strong points of the winner 
and flaunt the weaknesses of the losers. However, it is 
iT to the vendors and, in the long run, also unfair to 
B EDvospective user to establish the criteria for evaluation 
after the proposals are received. This kind of evaluation 
is subject to the vagaries of human nature, over which there 
is no control (Joslin Jo + 07 1971] . 

em. cost Only 
" This technique advocates selecting the lowest 
cost system that meets all of the mandatory requirements. 
However, what if the next-to-the-cheapest system is only 


Slightly more expensive than the cheapest one, and yet would 
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۳۰۰ perform it? Due to the unanswered cost and require- 
ment questions, the cost-only approach is rapidly losing 
favor, except for smaller systems with static workloads 
[Joslin 1977]. 

^ Any meaningful evaluation methodology should 
differentiate between mandatory and desirable features, 
Either a vendor shows that he can meet all the mandatory 
E rements cr his proposal is not considered for evalua- 
tion. If only one vendor can satisfy all of the mandatory 
requirements, he is automatically the selected vendor. 

Suppose, however, tnat three vendors were to 
satisfy the mandatory requirements, then the proposals of all 
three would be considered to be equally satisfactory. The 
purpose of the evaluation methodology is to establish some 
logical and defensible means of differentiating between the 
proposals of these satisfactory vendors and selecting the 
one that is best suited to meet the activity's needs. The 
items used to differentiate between the vendors who nave 
satisfied the mandatory requirements are desirable require- 
ments. Since there are many desirable features of varying 
importance, the evaluation methodology must find some method 
of establishing the relative values of these features and 
their relationship to the system cost, 
d. A Case History 
Company ABC wanted a computer system, The 

company needed to have a given set of problems processed 


within one hour's time, and- there were certain other 
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that had to be met. ABC sent out a request for‏ ۱65ب 
ET El, lo which five vendors responded. Three proposals‏ 
satisfied all requirements. The only significant difference‏ 
between the three proposals was the amount of time it would‏ 
take to process the problems and cost of the three different‏ 


ከከ ንገ መ The findings were: 


Vendor System Cost ($) Time to Complete Problems (Min) 


X 300, 000 De 
Y 215 000 Do 
Z 500,000 es 


Vendor Z was selected by the evaluators because 
they interpreted the time to process the problems in its 
reciprocal sense of how many sets of problems could be proc- 
essed per hour. Thus, vendor X could process 1.2 sets; 
Dex Y could process 1.1 sets; and vendor Z could process 
2.4 sets. They then divided each system's cost by the number 
of sets which the system could process, with the following 


results: 


Vendor Cost Per Set Per Hour 
X 250,000 
Y 260,000 
2 208, 333 


Ihe evaluators justified their choice by pointing 
out that vendor Z's system gave the most room for expansion 
and could process more sets of problems per dollar than either 


of the other systems [Auerbach 08080 
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2. sophisticated Techniques 


There are two basic evaluation methodologies which 
permit the evaluator to consider desirable features and 
establish the relative value of the desirable items. The 
approaches are weighted-scoring schemes and cost-value based 
approaches. 

a. Weighted-Scoring Technique 

Under this system, the prospective user preassigns 
Ine quantities of points to all items he considers impor- 
tant and then selects the system earning the most points. 

An example of this technique is shown in Table 2. Vendor B 
would be selected [Auerbach 1975]. 

Since this technique appears to satisfy all of 
the criteria listed, it may seem to be a good evaluation 
methodology. However, upon closer review it is found to fall 
down on the criterion which calls for "meaningful and under- 
standable" relative values between all desired items and on 
E her criterion which calls for incorporation of system's 
life costing. (See page 30 for the criterions.) 

It is difficult to establish a meaningful and 
understandable relationship between the number of points 
awarded for low cost. For instance, the example shows cost 
having a weight of 70%, but why TO rather than 30 or 50 or 
90$? The failing of this technique is that there is no 
common denominator among the items being weighted. Thus, 


there is no "meaningful and understandable" relationship. 
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AN EVALUATION USING WEIGHTED SCORING 





| ከ ከ 6 VendorsizScoz->3 
Evaluation Values 

Items % A B C 
Gost ae (O 60 50 
System potential 20 0 16 20 
Mechanical 
characteristics 2 E 4 5 
Vendor support 5 3 4 2 
METAS SCORE 100% 82 SL 80 
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Until a meaningful approach can be found to the proper 
distribution of points between the items desired, the 
weighted-scoring technique will mever be considered a very 
satisfactory evaluation technique. 
b. Cost/Effectiveness Ratio 

This technique is simply a subcategory of the 
weighted-scoring technique, except that with it, by dividing 
the total system cost by the sum of the points scored in the 
other desirable categories (effectiveness category), the 
prospective user can Select tne system with the lowest ratio 
of cost to effectiveness. However, such a division of 
points is generally not sufficient to establish a meaningíul 
relationship between cost and effectiveness {Joslin LOTT, 
Borovits 1975]. 

c. Cost-Value Technique 

None of the previous evaluation techniques 
proved very satisfactory under intensive investigation. 
Therefore, a new evaluation method, the cost-value technique, 
was developed in 1964, This technique combined the simplic- 
ity of the cost-only technique with the realism of the 
weighted-scoring technique. The result was a technique 
superior to both. It is superior to the cost-only technique 
because it considers some items in a computer system to be 
of value in addition to the system's cost and its compliance 
with the mandatory requirements; and it is superior to the 


weishted-scoring technique in that it establishes a meaningful 
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relationship between the items of value and the system's 
cost while at the same time incorporating system's life 
costing [Chorafas 1967, Joslin 1977]. 

The cost-value technique r2cognizes the necessity 
of evaluating the desirable features offered by the various 
computer systems proposed, With this method, the desirable 
features and the cost associated with the system are all 
that are evaluated; that is, the ability of the proposed 
Eus to perform the functions for which a computer is to 
be procured and the ability of the vendors to meet any other 
conditions specified as mandatory in the specification 
package are not evaluated, but are validated. If it is 
٠۰۰۰۰۰٦٦۹٢ the vendor or his system cannot perform as 
required, the proposal is eliminated from further considera- 
tion. With this technique a company can study any extra 
features offered in the proposals to determine whether the 
claimed extra features are important in themselves or are 
mere incidental elements that appear to be extra features. 
۲۰٠٠-116, a 60-nanosecond memory and a 10,000-card-a-minute 
card reader are not important features in themselves. More 
Important and desirable is the amount of slack time that 
exists in the proposed system on acccunt of these high-speed 
units. A study should be initiated to determine the value 
of every extra feature which is considered to be important, 

A distinguishing feature of the cost-value 
technique is the assignment of the value associated with the 


rable features in terms of cost, that is, dollars, of 
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۳۲ رت‎ 355182128 a cost value, in dollars, to the desir- 
able features cffered by the varicus vendors, a common 
Genominator is provided by which all offered desirable 
features may be related to each other and to the system's 
RREO Although the cost values assigned to the various 
desirable features still will be a matter oí each individual 
ከከ tion, and will continue to reflect the opinions of the 
EEENSHOPS, a value, when assigned, can be understood, examined, 
discussed, and changed independently of all other individual 
assigned values. 

An important benefit derived from the use of 
assignment by cost vaiue is that management can understand 
what is going into an evaluation, and is able to make 
informed decisions on the value of any disputed desirable 
features. The specific cost values established for each of 
the various desirable features found within each of the 
proposals are then used for the scoring of the proposals. 

In the cost-value technique, the proposals are scored or 
ranked by what will be referred to as a cost-value account- 
ing scheme. This is cost and value accounting, since 

some of the values and costs used, although stated in dollar 
terms, may nct involve real expenditures. 

The cost-value technique amounts to taking the 
total cost of a system proposed and then deducting the cost 
values of all the desired extras included in that proposal. 
The difference represents the derived cost of satisfying the 


mandavory requirements stated in the specification package. 
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The system having the lowest derived cost for satisfying the 
mandatory requirements becomes the system selected, since 
the values of desirable features offered already have been 
۱۳۳۲ into consideration in deriving this cost for satisfying 
the mandatory requirements. This ranking also can be lcoked 
at from a value-to-cost ratio, but the results will be the 
same if value is considered in its full sense, as value of 
mandatory requirements plus the value of the desirable 
features offered, and cost is considered to be the total 
cost of the package over the estimated life of the system 
[Joslin We77, Tatham 1969, Auerbach 1975). 1 
(1) Using The Cost-Value Technique. The cost- 
value technique's approach to the extra features (those 
proposed features above and beyond the mandatory require - 
ments likely to be offered by the vendors) is to appraise 
them to determine whether they are worthy of inclusion in 
the evaluation, and, if so, to determine the cost value of 
these features. To avoid any bias or appearance of bias on 
ስ ሙሉ ፡ ርዕ! the evaluators, and in order to be fair to both 
vendors and the user, this study preferably should be initi- 
ated before the proposals are received. However, it should 
be noted that the cost-value technigue actually is open- 
ended; that is, if any unexpected extra features are offered, 
they can be included as part of the selection, if deemed 
important enough, by simply assigning them their cost value. 
It thus becomes necessary to deal with either hypothetical 


Or realistically anticipated features. A sample listing of 
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features which may be corsidered in cases must first be 
established; then it can be seen how to go about assigning 
cost values to those items which really have value to a 
ection. Table 3 contains a sample listing of these 
features. 

۴۰۰٠٢٠٠۰٠٠۷٠٠٠٠۹ cost Ltems must be consid- 
ered in the evaluation. Items such as the cost of supplies 
۳ personnel may prove to be nondifferentiating in a given 
selection, but they should not be deleted from the evalua- 
۱۳۲ او‎ because, differentiating or nondifferentiating, 
they are still true costs associated with completing the 
ከ. ንክ ” o applications. 

Treating cost items as one-time costs or 
Pemeemuing costs is a matter of cataloging. The following 
rules must govern any proper treatment of cost items [Joslin 
1577]: 

1. The costs must be spread proportionately over the 
expected life of the system. 

2. The system costs must change to reflect the costs of 
any planned system expansion. 

For example, if the life of a system is set 
at six years and a uniform expansion rate of ten percentage 
per year is expected over the life of the system, then each 
ENDE continuing cost items on the list, if applicable to 
the yearly system cost, should be charged for six years. 
Thus it would be expected that the equipment cost for the 


sixth year would be larger than that for the second year. 
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TABLE 3-a 


Items To Be Considered 
In Computer Selection 


۱ Equipment Expansion Vendor's 
Charac- Potential ۱۰ +٣٢ 


teristics System 





qu 





TABLE 3-0 
CO 


Co 
+ 
UN 


ONE-TIME COSTS 


Site Preparation 


Electrical 
Air conditioning (cooling, heating, and humidity 
control) 
Power supply (including all wiring) 
Space for equipment 
Facilities (walls, ceiling, painting, draperies) 
False flooring (including bracings) 
Security provisions 


Equipment Installation 
Equipment Transportation (including insurance cost) 
"ስ ስሮ” Support 


Personnel (analysts, programmers, operators, instructors) 
Training (including transportation, living costs) 
Existing programs 

Backup facilities 

Machine time (checkout) 

Documentation 

Program and data conversion 


CONTINUING COSTS 


Procurement of Computer System Equipment (falls in one-time 
Pests category if system is purchased) 


Central processor and associated eguipment (console, 
۰۷۰ point Option, real time option, etc.) 
Peripheral computer eguipment: on-line or off-line 
remote-inquiry device, card reader, printer, etc.) 
Auxiliary eguipment 
Keypunch machines and other data-created devices 
ex immlcer steletype machine, etc.) 
Printers, sorters, collators, etc. 
Operation and Maintenance of All Electrical Eguipment 
Personnel (manager, analysts, programmers, operators, etc.) 
Program Development 
Supplies (magnetic tape, printer paper, cards, etc.) 


Indirect Cost for Space Used 
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TABLE 3-c 
SC UTE Pe CHARACTERISTICS 


SPEED 


| س‎ ini 


Time required to complete applications specified 


۶۰۰6۶089 
Add time (fixed and floating) 
Mult. time (fixed and floating) 
Divide time (fixed and floating) 
Move 
Other instructions (through all other instructions 
puente oan) 


Peripheral equipment 
Printer (lines per minute) 
Card reader (cards per minute) 
Card punch (cards per minute) 
Magnetic tape units (characters per secondj 
IAS (characters per second, average) 
Other equipment (through all other peripheral equipment 
listed 


9 ۳ 


Storage capacity of main memory (core) 

ር ቢይ capacity of immediate-access storage (IAS) 
Storage capacity of magnetic tape 

Characters per printed line 


NERT RTL, CC 
Program; tapes; cards 
O ABILITY 


Error detection; error correction techniques; mean time to 
failure, etc.; redundant components 


SPECIAL FEATURES 
Memory lockout; parallel processing 
PROBLEM TIMINGS 
ventral processor limited; input/output limited 
ENCICHABILITY 
Magnetic tape units; printers 
EATER CHARACTERISTICS 


Size of equipment (each piece considered); weight of 
equipment (each piece considered) 
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ABCE d 
EXPANSION POTENTIAL 
STACK TIME (amount of available free time on each piece of 
equipment) 
NFS Ll processor, magnetic tapes, immediate access storage, 
card punch, printer, remote terminals, etc. (through all 
other equipment offered) 
MAXIMUM EXPANSION (number of units that can be added to 
system 
Magnetic tapes, immediate access storage, card punch, printer, 
etc. (through all other system equipment offered), extra core, 


disk drives 


COMPATIBLE EQUIPMENT 
Larger processors 


Higher performance units 
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TABLE 3-e 
MANDO را‎ OS SYSTEM 


PROGRAM ASSISTANCE 

Development; writing; converting; emulating 
TRAINING 

Analysts; programmers; operators; managers; users 
MAINTENANCE OFFERED 
BACKUP AVAILABILITY 
PROGRAM TESTING 

NTS; Shift; location 
EXISTING SOFTWARE 

Operating system 


Fomeaumlers;impur/eucpuütr control; memory allocation; 
Sees 


peru; merge; system simulators or emulators; COBOL: 
FORTRAN; report generator; etc. 


DOCUMENTATION 
PERSONNEL LOANED 


Analysts; programmers; operators; users 
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Another important consideration relating to 
Bost items is that they should show the cost for individual 
E Res of equipment to be used. This should be true in all 
cases except when the system is to be used for less than one 
shift, or when the entire system is to be purchased. 

No cost items E be duplicative; that 
is, the system should not be charged twice for the same equip- 
ment or service. For example, if a card reader is used both 
online and offline, its full cost should not be shown twice. 
Similarly, program development, if performed by the user's 
ከን ን ን ን ር! rather than the contractor's personnel, and person- 
nel cost should not both be costed for this program. 

EQUIPMENT CHARACTERISTICS. The significance 
of the characteristics of any piece of equipment is measured 
in terms of the running time cf the system, which in turn 
determines the system's cost and expansion potential or its 
system responsiveness, Typical of the kind of equipment 
characteristics now being discussed are: the relative speeds 
SC capacities, hardware compatibility, switchability, relia- 
bility, and special features. For real time systems, these 
conditions usually will be stated as mandatory requirements 
[Chorafas 1967, Coutinho فا‎ | 

Sample problem timing items (times required 
to perform the benchmark problems) should not be evaluated. 
They should be used exclusively for validation or establish- 


ment of application timings quoted in the proposals. This 
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BEoddestion timing, in turn, is the base on which system 
cost and expansion potential are calculated. Since its 
value is felt in the other items, it should not be evaluated. 

EXPANSION POTENTIAL. The expansion poten- 
tial of a system is considered to be an important extra, 
since it allows for growth beyond the specified amount. Thus 
the system has the possibility of a longer life and of han- 
dling larger workload peaks. Another type of expansion which 
is sometimes important is the ability to add on different 
types of peripherals. 

VENDOR SUPPORT. Vendor's support is also 
deemed important to the cost-value technique. All of the 
items of vendor's support could be desirable features, since 
each offer could result in some actual saving to the user 
[Thrusse11 00 1 1972]. 

(2) Constructing Evaluation Templates. The = 
cost-value technique examines expansion potential by 
evaluating: 

l. The system's ability to handle additional workloads, 
and 
2. The system's ability to handle different peripherals. 

To evaluate the system's capability for 
nandling additional workloads, it is necessary to first 
calculate the run time required by the system to complete 
all required applications. The elements and aspects of a 
computer and its use that must be considered in any calcula- 


tion of the running time of a system are: 


دار 





days 


Speed 

a) Central processor 

b) Peripheral equipment 
c) Auxiliary equipment 
Capacity 

a) Central processor 

b) Peripheral equipment 
Special features 

a) Parallel processing 
b) Simultaneous operations 
c) Other 

Software efficiency 

a) Compiled languages 
b) Assembled languages 
Reliability 

a) Switchability 

b) Error detection and correction features 
Preparation time 

a) Setup/take-down 

5) Peogram insertion 

c) Media handling 
Nonproductive time 

a Reruns 

b) Program checkout 


If a system were used 24 hours a day for 30 


each month, it would be possible to get 720 hours a 


month of computer time. However, most manufacturers require 
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about 1% hours a day for preventive maintenance of the 
machine and this would leave about 675 hours a month. From 
this 675 hours must be subtracted various loss factors, such 
as unscheduled maintenance, idle time, setup time, machine 
malfunction time loss, program housekeeping, development and 
maintenance of programs, program errors, and cperator errors. 
These reductions cut the actual maximum time available for 
production to between 400 and 500 hours per month for most 
business applications and 500 to 600 hours per month for 
scientific applications [Josiin 1977, Webster and Johnson 
1976]. 

lhe figures for total production time 
available should be used to deduct the times computed tc 
Broeess the monthly workload. The time remaining, called 
slack time, is the time available for expansion. The amount 
of slack time available could be increased by reducing the 
time required to process the monthly workload, which could 
be achieved by adding processors of higher capability. 
However, more or faster units should be added only when the 
value of the additional slack time is greater than the cost 
of modifying the system to make this additional time avail- 
able. The worth of the additional slack time might be 
considered as the additional system life brought about by the 
m ۱1۵ 10۳۲ potential, The concept of system life applies not 
only to a purchased system, but also to a leased system where 
there is extensive investment in software and know-how which 
is geared to the existing system. Any changeover thus might 


ove costly. 
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YEARLY EXPANSION, The most meaningful way 
of preparing a value template for yearly expansion would be 
to look at the stated workload for each year, estimate the 
confidence level that the stated workload is correct, then 
increase the workload until a confidence level of about 95% 
is obtained. For example, if it were estimated that the 
stated workload for the first year was 100% of some base 
amount, after considering the case it mignt be found that 
there was only about an 80% confidence in that estimate. 
However, if that base amount were increased to 110% of the 
old base (having a confidence level of 80%), there would be 
88% confidence; if it were increased to 120% of the old base, 
there would be 96% confidence. The estimate would have to 
be increased to 125% of the old base before there would be 
100% confidence that the workload as then stated could not 
۳ ۱6 20۵60 im the firsteyear [Joslin 1977]. 

Assume that the system envisioned for this 
case was expected to lease for $100,000 a year. With these 
facts, the following value template might be established for 


the value of expansion for the first year: 


First Year Expansion Value Template 


Confidence Level 


Percentage Expansion Value 
5 20,000 
20 16,000 
10 8,000 
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The value figures are derived by saying 
that, if the user were willing to pay $100,000 to handle 
what he is only 80% confident represents the first year's 
workload, he should be willing to pay 25% more to have 100% 
confidence in the system's ability to handle all the first 
year's workload; that is a total of $120,000, or an increased 
value of $20,000. 

Similarly, to be 96% confident rather than 
80% he ought to be willing to pay 20% more, or $16,000, and 
so forth. In a similar way, an evaluation template could 
be prepared for each of the years, 

If the evaluation templates are surplied to 
the vendors, there should not be any need to adjust the 
vendors' proposals to reflect the greatest value for the 
user. If the vendors are not supplied with the evaluation 
templates, then the value of expansion potential must be 
calculated for each year. For example, if a vendor were to 
propose a system that was so modulated that every year his 
system took all the time available just to handle tne 
required workload, but examination of his equipment revealed 
that he could increase his system's capability by 10% for a 
yearly lease increase of $2,000, or 20% for an increase of 
$6,000, or 75% for an increase of $13,000, adjustments to 
his value of expansion potential should be made, 

If the previously established Value Template 
were to be used, the vendor's yearly values for expansion 


would be: 
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Increased Value Ratio To Increased Lease Price 


8,000 2 2006 
16,000 6,000 
20,000 $13,000 


The 0ی‎ SS ame ወ= cost Is $16,000 - $6,000 
= $10,000; a 20% confidence expansion is indicated. 

w 120107 03111222257 approach for determining 
the relative value of yearly expansion could make use of mar- 
EU utility analysis techniques, but similar results should 
be obtained [Webster and Jenason 1976. Thrussell 1976, Joslin 
1977]. 

EXPANSION BY NEW OR DIFFERENT PERIPHERALS. 
There are times when it is of definite value to be able to acd 
peripherals to the system that were not called for in the 
basic system requirements. For example, it might be possible 
to handle a given application without using immediate access 
storage. However, if the user feels that sometime in the 
future he might wish he had experience with immediate access 
storage (IAS), he might establish a value for having the 
system possess the capability of connecting as IAS unit. In 
fact, he might establish two values: the value of having IAS 
proposed and a lesser value of having the capability to add 
am TAS unit. 

The suggested method for determining the 
meme of such a capability is: 

l. Calculate the probability of needing the capability. 
2. Determine the cost of obtaining the capability inde- 
pendent of the present system. 


3. Take the product of these two figures. 
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lt must be remembered that there is likely 
to be considerable difference in tne evaluation of the 
various capabilities proposed, since no one application 
measures every possible capability. 

EXPANSION WITHIN A FAMILY. The advantage 
NIS type oi: expansion is that the programs written for 
one of the smaller computers ina family will run on the 
larger ones. Therefore, the only expansion cost is that of 
the new computer, not a reprogramming cost. To the extent 
thet this statement is true, the family aprroach could be 
۳ in a fashion similar to the extra system life approach. 
However, the inefficiencies of running programs on a large 
computer that were prepared for a smaller one must then be 
۰۰۰٢٣٢. 

EE Inere are several methods 
oí assigning cost values to vendor support items. The 
simplest method, for which the user cannot estimate the 
value of the support items, is to require the vendor to 
quote costs for various levels of performance. For example, 
if one vendor offered on-site maintenance while all the rest 
offered on-call maintenance, a feeling for the maximum value 
of the on-site maintenance could be ascertained by asking 
each of the other vendors to state the cost of such Service. 
sometimes, however, the cost quoted may be so excessive as 
not to make a fair base against which to award value. For 
example, if a user were impressed by some special program- 


ming routine and asked various vendors for the cost of 
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Empplyvinrg it, he might receive answers in hundreds of thou- 
sands of dollars, whereas if he himself were to go out and 
mm@ecure such a routine, he probably would not be willing tc 
pay over $5,000. In such a case, the $5,000 should become 
the base. Restated as a generalized rule: In cases where 
the user would place a value on a service lower than a 
vendor's cost, this value figure becomes the base for deter- 
۲۰۰٠۰۰ the item's value [Joslin 1971]. 

In some cases, the vendor may not be able 
۱ 0510© cost figures for supplying service equal to some of 
the levels desired, simply because he does not have the 
necessary facilities. In other cases, it might be practical 
only for the vendor himself to provide the service. An 
example of this is a special training requirement which might 
occur in a real time system, where some provision, probably 
a special program, must be provided to allow the trainees 
ESS vo tne remote consoles for training purposes, yet 
prevent their mistakes from destroying the good system. This 
kind of training aid probably can be provided only by the 
given vendor. In such cases, the cost value of such a 
service must be determined individually, and might be con- 
Siderably higher than the costs charged by any other vendor. 
But the higher cost-value figure should become the base. 

The cost value of these items also might be 
ascertained by the user, by taking each item in turn and 


determining its value to him. Among these cost-value items, 
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the one most closely representing the user's needs should be 
Rosen. However, the cost value should never exceed the 
cost of having the service contracted by someone else. 

Some items such as available backur and 
debugging facilities are support items on which the vendors 
cannot be asked to change or improve. Therefore, their cost 
value must be evaluated as the eens are proposed. An 
approach to determine the cost value of back-up would be to 
determine the probability of experiencing a catastrophic 
failure, then the cost associated with carrying on the 
computer activities on the back-up facilities available, 
he. cost times the probability of catastrophe should give 
the probable value for back-up of each of the various systems. 
دہ تک‎ determiration for debugging facilities could be 
handled in the same way [Chorafas 1967, Sabol 1972]. 

OTHER DESIRABLE FEATURES. Many other 
features might be considered in hardware selection. Items 
such as memory lockout or desirable compatibility can be 
handled by determining the cost that will be eliminated by 
۳5 inclusion of such abilities. Thus the costs that would 
nave to be paid to convert tapes of one kind to another 
would be saved if the two systems were compatible. This 
cost becomes the cost value. 

Being able to run a portion of the old 
programs on the new system is a desirable feature. Therefore, 
program compatibility (or portability) is another important 


aspect of compatibility. An estimate can be made of the cost 
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that would be incurred if that portion of the software had 
to be run elsewhere until rewritten. This estimated cost 
becomes the cost value of program compatibility. However, 
۱۳۲ ۵ compatibility is achieved through the use of an 
emulator or simulator and the resultant programs would not 
run at fhe efficiency of rewritten programs, then the value 
of this compatibility is decreased. The amount of the 
decrease would be dependent upon the frequency of use of the 
programs.  Infrequently used programs do not need to be as 
efficient as frequently used programs. 

Costs of the time and trouble that could be 
saved by inclusion of a memory lockout device become its 
cost value, which can be shown in an evaluation template 
[Auerbach 1975]. 

A system may be proposed that will enable 
management to have access to any information within the file 
in less than one minute, or to have management reports ready 
by 1:00 p.m. everyday. In such cases, a study must be 
initiated to determine the cost value to management of being 
able to have one-minute access, rather than ten-minute 
ENSSess as requested in specifications package, or the cost 
value of having the reports ready by 1:00 p.m., rather than 
3:00 p.m. as similarly requested. Where possible, these 
value assignments should be made in time to help the vendors 
With their bidding. 

With real time or time-shared systems, 


another area of desirable features should be considered. 
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For example, a time-shared system may call for eight remote 
nals with an inquiry from any terminal being handled 
within six seconds. Some of the systems proposed may be able 
to handle two or three times the required number of termin- 
als, The value of these extra terminals depends upon the 

ከ  ጥ ገቲኒርሃ of their profitable use or on some logic similar 
to that used in making the original decision that eight were 
required. The value of various speed responses should be 
determined and shown in an evaluation template. 

Another extra is the possibility of coming 
across an innovation or a new approach to the system. The 
prospective user could assign a cost value to such a new 
approach by making a realistic determination of the savings 
that are likely to accrue if he uses the suggested approach 
times the degree oí probability that the suggested apvroach 
will actually work, or by esvimating how much it would have 
cost him if he had had a special study made that might have 
come up with the same recommendation. 

Extras such as a purchase option offered or 
expected trade-in will or will not have value, depending 
upon the type of procurement plan to be used in the acquisi- 
tion of the system. 

(3) General Thoughts On The Cost-Value Technique. 
In applying the cost-value technique, the folloving points 
should be kept in mind: 
l. The methods described here for cost-value determinations 


are by no means the only ones that might be used. 
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2, The items chosen for discussion are not the only items 
Eun Of inclusion in a cost-value evaluation, nor will 
they all necessarily appear in any given selection. The 
circumstances of a specific selection determine the items to 
be used, 

۳ mere is nothing sacred in any of the cost values 
established, since the value of any item depends upon the 
۱۳۱۳۰۰۱ ۱۳۵۵0 of the user's need for that item. For instance, 
if the described system is to be used only for one or two 
applications and the size and volume of these applications 
are fixed then the cost value of expansion potential is 
likely to be nil. On the other hand, if the described 
system is the first system to be installed in a growin 
company, the cost value of expansion potential will be high 
because every hour of available expansion might be as valu- 
able as each hour in actual use. If the described system is 
to replace an existing, compatible system, some of the 
MEMO support items, such as personnel loaned or program 
assistance, may have no cost value. However, if the computer 
is for a relatively inexperienced group, such factors might 
have a cost value as high as, or higher than, $40,000 cer 
man-year [Joslin Kora 


Calling the items to be evaluated "extras" 





implies that the "extra" is the amount over the minimum 
acceptable or mandatory. The value of an extra nay be estab- 


lished independent of the proposals from preconceived values 
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which are created to show value for varying amounts of each 
of the extras evaluated. These preconceived ideas of worth 
are referred to as "evaluation templates". 

ESTABLISHING LIMITS. To assess the value 
emery given item is a difficult task. The logical starting 
place is what the item costs. If the item is competitively 
available, its value should never greatly exceed its cost. 
For example, if the cost of having a mathematical subroutine 
ስ by a software consultirg group is $10,000, then it 
would be reasonable for a user with little use for this 
routine to establish a value of only $500 for its 
ability. 

If only one vendor can supply a critical 
subroutine, its value is almost indeterminate. However, 
this case should not arise in cost-value assessment for two 
reasons: | 

۱ 11 the item is critical, it should be listed as manda- 
tory requirement and should not require value assessment. 

2. If the item can be procured from only one vendor, the 
full selection ought to have been handled as a sole-source 
procurement, again making value assessment unnecessary. 

DIMINISHING VALUES. The prevailing thought 
behind most of the evaluation templates created is that, as 
more of an item becomes available, the worth of that item 
decreases, This can be shown mathematically by exponential 


Curves such as these shown in Figure 2. However, for ease of 
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understanding, it is usually easier to break down each value 
EsskPenment into a group of smaller approximations [Chora fas 
1967, Tatham 1969, Sabol 1972, Joslin 1972]. 

Suppose that the ability to be able to 
expand by 20% is worth $20,000; by 40%, $32,000; by 60%, 
$40,000; by 80%, $46,000; and by anything over 100%, 
$50,000. An exponential curve could be fitted through these 
eames and the curve established, but it is usually not 
worth the effort. Using normal interpolation techniques 
between the defined points, the value of any expansion 
capability can be found. Thus, the value of having 50$ 


expansion capability could be found by taking: 





Difference between Difference in value 
10% and 50% 10% x for 40% and 50% 

Difference between 20% $8000 Difference in value 
40% and 60% for 40% and 60% 


The unknown difference is found to be $4,000. When this 
amount is added to the amount for 40%, the resultant value 
for 50% is found to be $36,000. Figure 3 shows this template 
plotted, using straight-line extrapolation between the 
defined points and using an exponential curve. The value 
for 50% expansion, if taken from the exponential curve, would 
be approximately $37,000. | 
Am explanation nas been given of some c 
niques for determining the cost values of a number of items 


that should be included in any selection. The cost values 


derived for the various vendors are applied as credits to 
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True Exponential Shape of Normal Evaluation Templates 


Required A B 
Features | | | 


Value (X: 20000) 





0% 20% 46% bO% 80% 100% 
Extra Features 


PIGURE 3 
Evaluation Templates 


B 
| 


b5e K — | سم‎ 
اا‎ 
uU" 
R ب‎ 
AA T 
١ 1 


20% 40% 60% 50% 


IP 


64 


i 


ag a ነ 
ሰዋ | 


ISP 


E 
Ww 
u 
wi 


E 





EPfset the costs of the system and the proposed services. 
The vendor whose proposal shows the smallest difference in 
But-of-pocket costs minus credits is the one to whom the 
Esntract should be awarded- 

d. Requirements-Costing Technique 

This technique is conceptually the same as the 
cost-value technique, only under this approach a vendor is 
assessed a preestablished dollar value or worth for ae 
desirable feature not offered (or offered at a cost that 
exceeds its worth) by the vendor in its proposal; or if the 
vendor offers the feature, but at some charge, then the 
vendor is assessed that charge. The system selected is the 
one having the lowest overall total cost (including not only 
the cost of the vendor's hardware, software, and services, 
Fu ISO other costs for such items as staffing, power, air 
conditioning, etc., and assessments for features nct offered). 
An example of requirements costing is shown in Table 4. 

The requirements-costing technique and the cost- 
value technique are essentially identical, and they prove 
exceptionally satisfactory once the dollér values of the 
desirable features are established. Both of these techniques 
meet all the criteria listed as essential for a superior 
evaluation methodology. 

e. Dynamic Approach 

The problem and an approach to a solution is 

presented here in the form of an example. Assume an organi- 


zation which has decided to replace its existing computer 


65 


ae nn:‏ وہ یی سس 
4 
5 





u 
۱ = 
u | 
| 
u 
Ka 
۲ 
u 
5 
G 
ከ 
» 5 
DS 
ጠለ 
۱ i 
5 l 
ዎጆ 





TABLE 4 


AN EVALUATION USING REQUIREMENTS COSTING TECHNIGUE 


Maximum Vendors' Cost $ 
Evaluation Values 
Items $ A B 6 
Vendor Costs IODO 1 20040000 9 90 
Other Costs 0100/0106 02,000 90,000 
Assessments 
System 400,000 240,000 100,000 20,000 
Potential 
Technical 200,000 60,000 35,000 70,008 
Characteristics 
Vendor 50,000 104000 20 0 
support 
Total Cost ------- goo 1,455,000 15442205000 


Auerbach 1975 
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system in preparation for a major expansion in its informa- 
tion processing activity, proposals are on hand from three 
manufacturers, A, B, and C, each of whom has a number of 
systems to offer, named Al, A2,...C4 and benchmarks have been 
performed for a representative sample of the organization's 
workload on one or more systems of each manufacturer, 

The least powerful system, Al, has been chosen 
as a reference point, and its capacity assigned a valve of 
l. Based on the benchmark runs and extrapolations from them, 
tne capacities of the remaining systems have been determined 
as multiples of the capacity of Al and tabulated as in Table 
5-a, where each column represents systems of comparable 
capacity. From the table, it follows that system 42 of 
manufacturer A is 1.8 times more powerful than system Al, 
Ed so on. 

The rental prices of the various systems detailed 
above, E dihe software charges, are shown in Table 5-b, 
Dividing the data of Table 5-a by those of Table 5-b provides 
۲٣٠٦۰٦۰٣٢۹ 210۵۳ of the capacity per dollar outlay, which can 
be called cost-effectiveness, for each of the systems. These 
figures are shown in Table 5-c. Assume that the firm faces 
an anticipated growth in workload as shown in Figure 4, Year 
O on the horizontal axis is the current year, and year i is 
the year of installation. The planning horizon considered 
is six years. The vertical axis represents anticipated 
workload expressed as multiples of the workload capacity of 


the reference system, Al. The planned major expansion in 
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Manufacturer 


Manufacturer 


Manufacturer 


TABLE 5-a 


1 2 3 
1.8 SENG 
کک‎ በ 2.3 
TABLE 5-5 
System 
Jj 2 3 
560 $82 $100 
$77 --- ---- 
--- $82 $98 
TABLE 5-c 
system 
1 2 3 
12.67 2.20 2.60 
1.56 2: መ 
---- 1a .ئ2‎ 


Ein-Dor 1977 
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marormation processing activity causes a rapid climb in the 
curve in years one through five with a subsequent return to 
EEsbility in year six. 

At the end of six years, for this system, the 
workload will be about three times the capacity of the 
reference system. Under the conventional form cf computer 
selection, a system should be chosen which either satisfies 
immediate requirements and is evaluated as having good 
growth potential or satisfies requirements for the next five 
or six years. The only systems which meet this requirement 
are A4, BYU, and CA, This is illustrated on the right-hand 
margin of Figure 4. Since, from Table 5-b, system B4 is both 
the cheapest of the relevant systems and also has the highest 
figure of merit for cost effectiveness (see Table 5-c), it is 
the logical choice in this case. 

The solution can be improved by using a modified 
approach to the upgrading of computer systems. It is gener- 
ally conceded that there will be no more revolutionary change- 
mers between [generations of computers, and evolutionary 
growth will become the order of the day. With respect to 
peripheral equipment, the evolutionary approach is already 
well established and most systems are progressively upgraded 
by the addition of new peripherals, or the exchange of lower 
ት 1! ርሃ peripheral units for those of higher capacity. This 
is especially obvious with respect to discs and other mass 


Storage units, 


TO 





Given the compatibility provided between the 
processors and operating systems within the product line 
marketed by each manufacturer, it is now feasible to apply 
the evolutionary approach to cpu's as well as to peripherals. 
It is then possible to plan the replacement of a cpu after 
one or two years! service rather than after five or six 
years, If this approach is accepted, then one no longer 
selects a single computer system but rather one selects a 
series of computers within the compatible range offered by 
one manufacturer. 

Assume that it is feasible to install a system 
for as little as one year, provided that it will be replaced 
by a compatible system from the same manufacturer's product 
line. Assume further that all installations are performed 
at the beginnings of years, and that system must be upgraded 
at the beginnings of years in which they would otherwise 
become saturated [Ein-Dor 1971]. Depending on these assump- 
tions the growth path for each of the systems would be as in 
Table 6, and as exhibited in Figure 4. 

lhe discounted present value of the rental 
systems, assuming a 20% cost of capital, is $3.15 million for 
manufacturer A, $3.50 million for B, and $3.78 million for C. 
Thus the dynamic evaluation presents A as the economically 
imal Solution rather than B in the conventional evaluation. 
The total undiscounted cash flow for this solution is $6.65 


million compared to $6.98 for the static selection procedure. 
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The analysis above has been performed on the 
assumption that equipment is to be rented. The method is 
Ata lly valid for purchase with purchase prices appearing in 
the analysis instead of rentals. Furthermore, if the dynamic 
- analysis is performed for both lease and purchase, it can be 
of assistance in deciding on the method of acquisition. In 
using this approach for selecting the form of acquisition, 
one must take care to ensure that the values incorporated in 
the analysis are comparable. This requires that all relevant 
costs be factored in. For instance, where maintenance charges 
are included in rentals, they should either be added to pur- 
chase cost also or else rentals should be computed without 
mainterance. If purchased equipment is to be replaced, resale 
values should be determined and treated as negative costs. 
Any tax or excise differentials shculd also be taken into 
consideration. 

Because of varying life expectancies of units 
within the projected system growth path, and because of 
Meariations in the ratio of purchase prices to rentals, it is 
most certain that an economically optimal solution will 
indica te that some units should be purchased and others 
leased. This implies that the analysis should be performed 
Stepwise. As each change is made to the system in the course 
of the analysis, the profitability of lease versus purchase 
should be evaluated; new units should thereafter be consid- 


ered as acquired in the least cost mode. 
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This methodology is a basis for determining the 
relative cost-effectiveness of product lines in a given 
ration, This alone is not, of course, the only criterion 
for system selection. Other criteria such as software 
availability, manufacturer service, or reliability may well 
be at least as important, However, it is also wise to base 
one's decisions on as accurate a determination as possible 
3 9۳ cost factor, 

Ihe steps involved in the dynamic evaluation of 
cost-effectiveness overtime for a series of computer systems 
15 as follows: 

1. Determine relative capacities of relevant systems; 
possibly by benchmark runs. 

2. Forecast the workload for the planning period in terms 
of a reference systen. 

3. Prepare a growth path for the systems proposed by each 
manufacturer with respect to the forecast workloed and deter- 
mine rental (or purchase) costs for each system. 

4, Determine the discounted current value of each outlay 
stream, and so determine which manufacturer has the most 
cost-effective product line for the situation under study. 

The application of this method may lead to 
considerably different results than the conventional methods 
ENNcost comparison. It determines a unique solution for a 


unigue situation which is not generally transferable, 
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f. Present Value Analysis 

A problem facing all buyers of computer equip- 
ment is deciding which system will cost the least and bene- 
fit his company the most. To determine if an investment in 
a proposed system will enhance the current value of a firm, 
present value analysis may be used. The present value of an 
amount to be received in the future is the equivalent value 
today of that future sun. 

Because of the investment alternatives, future 
receipts should be discounted (their face value should be 
reduced) to an equivalent present value if they are to be 
compared with present receipts. For example, the present 
value of $126.25 to be received at the end of four years has 
a present value of only $100 if the investor has an opportun- 
ity to earn six percentage on invested funds. That is, $100 
invested today at six percentage compounded annually will 
accumulate to $126.25 in four years. 

Generalizing, the present value (PV) of a future 


sum can be determined using the equation 


y _ FV, 
M bel (1+K)® 


where FV is the future sum, K is the opporturity cost (or 
rate of return), t is the year the future sum is received or 
paid and N is the number of receipts. 

Since the decision to invest in a system should 


be considered independently of the method used to finance it, 


(2 





the system should be evaluated as if it were being purchased 
for cash. The net present value (NPV) of the proposed 
System should be determined by discounting all the incremen- 
tal, after-tax cash flows associated witn it, using the 
000015 cost of capital as the discount rate. The cost of 
55.7٦31 is the cost of new funds required to replenish the 
cash used for the purchese of the system. This process is 


expressed by the equation 


+ m 


NPV <= Ds (Re -C,) Gen ; š 


C 
t+1 (1+K) (1+K 


where; 

Ry: added gross revenue generated from the system, 

C+: the added cost of operating the system (operating 
ES do not include any financing cost such as interest or 
lease cost), 

۲ the firm's tax rate, 

De: the depreciation, 

t: the time period, 
N: the number of years the asset will be economically 
useful, 

E the salvage value, 

BVy: the book value, 
۳ the initial purchase price, 
K: the تا‎ COSC OL capital. 
If the NPV ís greater than zero, the system will 


add value to the firm because its rate of return is greater 
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En the cost oí funds, and therefore it is a desirable 
project. When two systems are being considered, the one with 
the larger NPV should be chosen. If two alternative systems 
provide the same service or revenue (R+), this item may be 
assigned a value of zero in the equation of NPV. In this 
case the NPV becomes negative and the system with the NPV 
closest to zero (the less costly alternative) should be 
selected. 

Ihe relevant benefits provided by the asset are 
the net cash inflows, which are discounted at the firm's 
Seago Ct Capital to obtain their present value. If the 
present value exceeds the cost, the asset is financially 
desirable because it adds value to the firm [Roenfelt and 
Fleck 1976, Szatrowski 1976] . 
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የ ከመን ን ከህ WORKLOAD DESCRIPTION 


The most important part of any system specification is 
the part which describes the type and amount of workload to 
be run on the system. Performance is the degree to which a 
computing system meets the expectations of the person 
involved with it [Doherty 1970]. Performance is a reaction 
of a system to a specific workload. It is, therefore, 
essential that the right workload is used when evaluating 
the system and that the workload characterization is suffi- 
ciently representative to account for all significant 
factors. A good workload description Shculd serve three 
ሽፕ ስት ለ ገኔ functions: 

l. It should permit the vendors to determine what they 
need to propose for automatic data processing equipment and 
software (ADPE/S) to satisfy the workload requirements. 

۳ should facilitate the verification of the proposed 
systems, both as to their capabilities to handle the work- 
load, and as to the time required to complete the workload. 

=» it should permit realistic costing of the bid systems. 

۱ ۱ ا‎ two points, permitting determination of the 
proper system by the vendor and the verification of that 
system's capabilities, can be combined. If a good technique 
is achieved for the second purpose, that same method of 


workload description will also serve the first purpose, 
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User's applications, once translated into programs and 
commands, can be characterized by the type and the amount of 
resources the system will have to allocate to execute these 
programs and commands. The total of resource demands gener- 
ated by the user community represents the system workload 
[Svobodova ۱۳ 0 6 ld Examples of parameters used 
to describe computer system workload are presented in Table 7. 

In many computer installations, the instantaneous work- 
load changes quite unpredictably. This is especially true 
for interactive systems. The speed of the user's response 
ERES an important role in what load is generated at individ- 
ual system entry points; this humar factor only enhances the 
۱۳ 9501 3 2111 37 of workload changes. It is this uncontrol- 
lable fluctuation of the system workload that makes the 
۳۰۰٦٠٠ tion of system performance so difficult. 

Generally, the workload of a computer system nas certain 
"2035151131 ee do not change over reasonably 
Tong time periods. It is then possible to: 

l. Characterize the workload by distributions of demands 
made on individual system resources. 

2. Define a unit of work and express the workload as a 
member of such units. 

Quantification of workload by work units is used when 
defining and comparing system processing capabilities. A 
unit of work is assumed to require a fixed but not necessar- 
Dey explicitly known quantity of computation. Generally, 


lt is very difficult to define a unit of work. Even when 
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EXAMPLES 


TABLET 


OF WORKLOAD PARAMETERS 





Workload Parameters 
Job CPU time 

Job I/O requests 
eeu service time 
I/O service time 
Entererrival time 
۲٢٢ ዕዝ 

mocked time 

Memory requests 
Working set size 


Locality of reference 


User response time 


Ber intensity 
Number of simultaneous 
users 


Number in the system 


Instruction mix 


Description 


Total CPU time requested by a 
single job 


Total number of I/O operations 
rues ted by a Single job 


CPU time requested to process a 
single CPU task 


I/O time required to process a 
single I/O task 


Time between two successive requests 
for a systen service 


۶۲۰٠٦۹۶۰٠3۰67۰3 5512۲۰64 to a ۵ 
user 


Time a job is incapable of receiving 
CPU service 


Amount cf memory requested by a 
single job 


Number of pages of a single job that 
must be kept in the main memory 


Time for which all memory references 
made by a single job remain within 
a single page or a set of pages 


lime needed by a user at an inter- 
active terminal to generate a new 
Deauest (chin and type time! 


Processing time per request/user 
response time 


Number of interactive users logged 
21 7777 


Number of jobs or tasks being serviced 
or waiting in gueues for system resources 


Relative frequencies of different tyres 
of instructions the system must execute 





Svobodova 1976 
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programs are broken down to very elementary operations (NS 
instructions), the characteristics of such elementary opera- 
ons differ. 

A workload model serves as a workload of a real computer 
System during performance measurement experiments or as an 
Eu to a model of the evaluated system. The purpose of 
using workload models is to: 

1. Provide representative workloads for comparative 
Eweormance evaluation of different systems. 

2. Provide a controllable envircnment for experimental 
pertormance optimization studies. 

3. Reduce the quantity of data that have to be analyzed. 

4, Present the system workload in a form required by a 
system model. 

Alternate choices in system configuration and algoritams 
and the effect of different control parameters must be evalu- 
ated for the same workload. Generally, only one alternative 
can be examined at a time, thus requiring that the workload 
used as the input to the system during evaluation be 
reproducible. 

Ihe characterization of workload by demands made on 
system resources can also be used to define a unit of work. 
In fact, the workload parameters given in Table 7 are already 
defined with respect to a specific logical unit processed by 
a computer system. Such a logical unit is often adopted as 


a unit of work. To satisfy the requirement that a unit of 
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work represents a fixed quantity of computation, this logical 
unit is afixed with characteristics representing the mean of 
characteristics of all such units processed by the system. 
The real system workload, that is, the workload generated 
Becher user community in the normal production environment, 
is generally not reproducible in its exact composition. 
However, if the statistical properties of the system work- 
load do not change with time, the workload is statistically 
reproducible. The real workload can be used to drive the 
system during evaluation experiments, but the measurement 
intervals must be sufficiently long, and it is necessary to 
collect and analyze large amounts of data to ensure that the 
statistics are correct. The minimum measurement interval 
may range from minutes or hours if the system workload does 
not change with the time of day, to weeks or months if the 
۱۱ ۰1 99 0 exhibits significant changes with such periodicity. 
System workload may remain stationary for quite long 
ECS Of time, but in general, its characteristics change 
slowly as the user community changes because new applica- 
Buena are added and the old discontinued. In addition, the 
user community tends to adapt to system changes, and as the 
users change their habits, workload characteristics change. 
Thus in a long term, the real workload is not reproducible, 
System workload is characterized by demands for system 
resources. Ideally, a workload model will have the same 
characteristics as the real workload. The model is accepted 


as being representative of the real workload if its application 
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results in the same steady state performance [Ferrari 1972, 
Svobodova 1976]. The lack of proper understanding of work- 
Sad characteristics is a serious obstacle when the goal is 
E redict effects of system changes and design alternatives 
EN Srformance. Extensive empirical studies of programs may 
reveal many interesting properties that should be considered 
Mine the initial system design. It is also important to 
study the habits of system users. Performance effects of 
certain system changes measured against a once representative 
workload model may be positive, yet in reality, the users may 
react to these changes in such a way that the overall effect 
will be negative. The true behavior of the eventual users 
may be quite different from the behavior assumed for the 
purpose of system selection or design [Warner 1972]. 

A system that is too carefully tuned to a specific 
projected workload might not meet the performance objectives 
መጠይ ስ real workload turns out to have different character- 
istics. It is thus necessary to have a means of examining 
performance in the lisht of different workloads. Flexibility 
mea controllability of workload characteristics is an important 


property of a workload model. 


A. INSTRUCTION MIX 

An instruction mix represents the relative frequencies of 
@itferent types of instructions a system must execute during a 
specified interval of time. The instruction mix specifies rela- 


Give usage of different types of instructions in a particular 
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En በ1501 since each instruction may require a different 
time to execute, performance (instruction execution rate) of 
an instruction set processor can be evaluated with respect 
to the requested instruction mix. Instruction mix is used 
۳ vo main areas: 

l. Selection of computer hardware, 

meee Design of new processors. 

۱۳۲ ۲۱ first case, the typical instruction mix for the 
class of applications planned for the system must be defined 
such that it can be used across a wide range of different 
۳ ction sets. That is, a typical instruction mix speci- 
fies frequencies of different functions, ravher than actual 
meeeructions that perform these functions. A typical mix 
might be in the proportion of five adds, two compares, one 
۴ ۱ واه‎ one multiply. This, then, might be described as a 
Max Of instructions. By multiplying the frequency for which 
۳ instruction is typically used by the time a particular 
machine takes to perform the instruction and adding tnese 
meeewner for the instruction mix, one can arrive at a figure 
which represents the time for the instruction mix on the 
particular machine. This figure can be calculated for vari- 
ous machines and thereby the machines compared for an instruc- 
sion mix appropriate to a particular type of job. These 
figures can be NEL at from the theoretical timings for 
instructions of the machine or they can actually be measured 
by running a mix containing the appropriate number of 


instructions on the machine. 
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ፈቄ የ 1 1-1011 times are in milliseconds or microseconds, 
RE is convenient to run the mix, maybe, ten thousand times 
in loop consecutively so that the start and finish can be 
measured in minutes on 8 stop watch. Some of these mixes 
have been commonly adopted as means of comparison. The most 
የንግ መሮሠ1]ሇ ህይፎቧጠ instruction mix is the Gibson mix, which can 
be classified as a "general purpose" mix. Figure 5 shows 
the calculations of the mix time and Figure 6 shows the 
approximate Gibson mix times in milliseconds for a number of 
۳ ۹٠ 

Instruction mixes are also a means of comparing tne speed 
Soins arithmetic in machines. Even so, this simple tormo. 
comparison may be prejudiced by dissimilarities between hard- 
ware which, perhaps, are advantageous to one machine and not 
the other. For example, word length varies between machines 
and to the user this is of some importance from the point of 
۱۳۲ 01 accuracy; i.e., in a process control application ít 
may be perfectly adequate for the hardware to handle only 
four decimal digits, however for numerical analysis applications 
ten decimal digits may be required. Quite obviously a simple 
comparison of arithmetic operation is valuable only with other 
۳۰۶۱13٥1061 about the specific applications [Graham and Yearsley 
1973, Svobodova 1976, Gibson 1970, Joslin 1977]. 

Fnecruetion mix depends on many factors that are diffi- 
cult to account for, such as the number of operands per 
instruction or different addressing modes. Due to these 


factors, the number of instructions needed to run the same 
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CALCULATION OF MIX TIME 


INSTRUCTION MILLISECONDS SECONDS 
5 adds 10 50 
2 compares 5 10 
Multiply 10 10 
il subtract 50 50 
ከየ en T 0.120 
FIGURE 6 


APPROXIMATE PERFORMANCE FIGURES 


PROCESSORS GIBSON MIX MILLISECONDS 
ATLAS ques 

7094 NS 

1106 0.24 

360/65 0.17 

1108 0 

360/75 CUM 

1906A 0.10 

6600 0709 

11100 0.025 
370/195 ONSE 

7600 2 

370/165 ۶ھ‎ پ7٦‎ 
370/155 puo Tue e 707٤7 
370/115 Approx. 0.30 


Yearsley 1973 
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task on different machines may vary significantly. Also the 
mascruction mix is dependent on the programming language in 
Een the application is coded, the translator of this 


language, and finally the programmer [Lunde OTA, ovo bOdova 


1976] . 


B. BENCHMARK PROGRAMS 

A benchmark is defined as "a point of reference from 
which measurements can be made" [54253 Loma A benchmark 
can be an instruction, a special program or a sequence of 
calls to selected software components. In most cases, 
however, the term benchmark is used to mean a job or a set 
of jobs that represent a typical workload of the evaluated 
system. Benchmarks play the role of a drive workload in the 
real system, both for the purpose of comparative evaluation 
of different systems and performance optimization. A good 
benchmark will exercise all system ren (job scheduling, 
file management, I/O support, language processor, etc.) ina 
manner in which these functions are used or are expected to 
be used in the actual production environment. 

A benchmark representative of the current system workload 
can be assembled from already existing programs. Jobs to be 
included in the benchmark may be selected by random sampling 
of the job stream. This method does nct require an expiicit 
ፎክ ጉ ርገረጠረሮር of characteristics of individual jobs, but it is 
then difficult to determine how many of these randomly- 


selected jobs must be included in the benchmarks [Shore 1970]. 
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The real system workload generally consists of several 
Meses of applications (scientific problems, payroll, file 
update, etc.). A benchmark or, as it is sometimes called, a 
benchmark mix, can be constructed as a properly weighted mix 
ST Jobs representative of each class. However, demand 
characteristics of jobs performing different functions may 
greatly overlap. 

۱۱۳ most rigorous approach rests cn partitioning jobs 
mmeorclasses according vo their characteristics. The job 
wich characteristics closest to the typical characteristics 
for its class is selected to represert the class in the 
Benchmark. A selected job is assigned weights proportional 
to the percentage of workload that falls into that same 
Bacegory. Partitioning of jobs according to their true 
characteristics can be accomplished by cluster analysis. A 
clustering algorithm assigns jobs to a predetermined number 
of groups called clusters such that the differences between 
members of the same cluster are small compared to differences 
between numbers of different clusters, 

A benchmark constructed from real jobs is apt to be 
System dependent. In general, such benchmark is not directly 
usable as a drive workload of a different system. A consid- 
erable conversion effort may be necessary to create a bench- 
Rm (or several different systems [Joslin 1. ከከ ሪር). یی‎ Ya 


1976, Joslin 1965, Rosen 1976, Hunt, Diehr and Garnatz ۳ 
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There are several Steps to obtaining the mix of represen- 
tative programs to be used for benchmarking purposes, The 
following general guidelines should be kept in mind while 
Searching out representative benchmark programs: 

l. Where possible, benchmark programs should be written 
in a standard higher-level programming language; e.g., ANSI 
FORTRAN or ANSI COBOL. 

2, The mix of benchmark problems should be small Been 
that it is capable of being processed during e single half- 
day benchmark demonstration. 

3. ihe selected mix of benehmarks will demonstrate that 
the supplier's proposed system contains adequate memory and 
input/output devices, that the software proposed is opera- 
tive and adequate, and that it has sufficient throughput 
Speeds to the normal wcrkload. 

Ihe benehmark programs are not to be selected to prove 
the worst case situation, but rather to demonstrate 5 
۱۳ Capability for normal situation. If it is necessary to 
assure capability to handle worst case situations, benchmark 
programs selected for that purpose will be obtained; but 
tney are rot to be included in the mix; rather they will be 
treated separately as capability benchmarks. 

lhe results of the benchmark will help in the evaluation 
K Ort by: 

1. Proving that the vendor has a deliverable compiler and 


operating syStem. 
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۳۰۰۰۰۰۰۱٦ 7٣٦26۳6 evaluator an opportunity to compare the 
relative speeds of the different compilers. 

3. Determining the relative efficiency of the generated 
ብክ ከ code by comparing results of execution of the compet- 
mie Object programs. 

IL. ring the evaluator sufficient information to allow 
determination of minimum internal memory requirements, 

ies fast calculation is based on the size of the 
largest program which must be memory-contained at any one 
time, and is derived from: 

1. The benchmark results, which allow determination of the 
Bacio of object language instructions to Procedure Oriented 
Language (POL) statements, and the averege size in terms of 
memory locations of the object instructions. 

2. A user-generated estimate (in POL statements) of the 
Size of the largest program to be core-contained. It is 
necessary to calculate the memory required for the program 
(MP) from the formula: MP = R x NS x IS, where: 

R = retio of object language to POL statements (from the 
benchmark), 

۱۲ < (estimated) number of POL statements for the largest 
program, and 

IS = average instruction size determined from the benchmark 
by dividing the memory required by the number of object 
Ka Cructions. 


Ihe estimate of required internal memory is completed by 
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adding the program requirements, the memory requirements for 
the operating system resident, data and buffer storage, and 
communications-oriented subroutines. 


۳ ۰۱۳۳01۵ multiplication of the following factors: 


benchmark object code 5 _ estimated object 
benchmark POL x estimated POL = code 


4. An estimate of the size of the average object instruc- 
۳ This is obtained from the benchmark by dividing its 
memory need by the total number of object language statements. 

Estimate of core for resident supervisor input, cutput 
buffers, and communications-criented subroutines. 

The summation of 4 and 5 will give an estimate of the 
required internal memory [Rubin ilr E 
۱ ۳۳1 7۵ 10۳ of Representative Programs 
Ihe following paragraphs describe a method for 
obtaining tne representative programs. 
a. Application 
List each of the applications making up the 
total workload. This is illustrated in Table 8. 
DEENECOsPams and Tasks 
For each computer program pertaining to the 
above applications, list the program and provide the infor- 
mation required in Table 8. For new programs or for 
acquisition of equipment that is for a new installation, it 
will be necessary to go through the normal design process 
with program flowcharts which lead to estimates cf program 
Bun Cimes, or to simulate the programs to obtain this infor- 


mation. Once the necessary estimates are obtained for 


91 





TABLE 8 


APPLICATION, PROGRAM AND TASK INFORMATION 


٣۰۰۰ 0 Todi vidual Éstimated 
ama Program Run Time Frequency Monthly 
Tasks (hours) Per Month Time (hrs) Subtotal 
EES (ዛ 
Application Á 
ADMINISTRATION 
۶۲۰ Time /Cost 
Studies 2.92 22.0 11.44 
E Sort 19.09 
E Edit 1.44 
E Payroll 
Processing 0.25 473 MOS 
۳ Sort 100 
b, Validate 0.08 
A3: Transportation 
Usage 9.30 
a. Compile - 
Cobol 0518, PO 2258 
D Extract 0.40 po 8.80 
:27م‎ Miscellaneous 20,08 
E Sort 2702090 
b. Update 10.09 
Application B 
EXSTHIBUTION CONTROL 
St: Mentor y 
87٦ھ۶۶‎ 7 0.45 220 18912 1010) 
ER Sort 19 
Be: Allowance 
Generation DES 
a. Compile - 
Cobol EOD 16 0250 
BD, Extract 12,00 1/6 ፡ 11019) 
E Compute 19.00 
Application K 
MATRIX INVERSION 
Time Required on System Total 880.00 


Joslin 1977 
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Table 8, these new programs can be treated in the same way 
ES any existing programs. Each program is broken down into 
RE major functions or asks, such as, sort, validate, update, 
۳۲ 2 واه‎ compute, card-to-tape conversicn, tape-to-printer 
Menversion, trajectory calculation, simulation, matrix 
meoroulation, etc. 

c. Task Summary 

From each of the programs listed in Table 8 
extract similar tasks and prepare a Task Summary Sheet for 
each task (see Table 9). Provide the information required 
in accordance with table headings which are explained below, 
nica is the description of the columns in the Task Summary 
sheet, 

PORN EE per the code 
for each program in which the task is found. In the exarple 
shown in Table 8 the identification codes which would be 
given on a Sort Task Summary Sheet would be Ala, A2a,...A27a 
ERG Bla, etc, 

iW enon FILE DESCRIRTI TiS Section is divided 
EGO four parts: 

1. Media Code. Enter a mnemonic for the media that 
contains the I/O or file. Examples are: 
MT Magnetic Tape 
PT Paper Tape 
PC Punched Cards 


PR -Printer 


Do 





TABLE 9 


TASK SUMMARY SHEET - DESIGN 





The column headings are successively: 
Identification 
RO or file description 
Media code 
Number of devices 
Category 
Brock size 
Monthly averages 
Frequency 
Volume 
Total time (hours) 
Peripheral equipment time 
Magnetic tape 
Card 
minter 
Cther 
Internal storage requirements (in K's of storage) 
Language 
Present 


Planned 


e 





2. Number of Devices. Number of devices that will be 
required for the use of this media which will have the same 
capability and category of use. 

3. Category. Code designating tne type or use of the 
WO or file. The following codes shall be used: 

O Sounece or ori gimal inpuk 

1 Master file 

2 Intermediate, working or scratch 
Si inal output 

۲ ርር Size, Product of the number of characters per 
record and records per block. 

MONEY. Tais colina is devided into Tour parts: 

l. Frequency. Give the monthly run frequency of this 
program, 

2. Volume. The number of blocks contained in the I/O or 
file. If this is a multi-tape file, follow the number of 
blocks by a slash and give the number of tapes. The volume 
to be recorded will be the average per unit, per month, for 
phis task. 

3. Total Time. Average total time to perform this task 
in the identified program. All times shall be given in hours 
ama” hundredths of hours. 

4. Peripheral Equipment Time. Estimated average time 
required per task by each type of peripheral equipment. If 
Bemilar units of differing capability are used. this timing 
information should be based on the highest capability avail- 
able. Due to simultaneity and overlap, it is not expected 
that the total of the individual units will agree with the 


total time. 
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INTERNAL STORAGE, Estimated amount of internal 
storage required to process the task (not the full program, 
me the program is a multi-task program). If two or more 
processors are used for the task, enter the information 
mecordingly. 

LANGUAGE. List the language in which the task 
is programmed (first column) or is to be programmed (second 
column). If task is a self-contained library routine, the 
initials "L.R." should be entered. 

TOTAL TASK TIME. At the end cf the lest Task 
Summary Sheet used for each type of task, there should be a 
۱۳۳ for total task time. The sum of these totals from ail 
the Task Summary Sheets should equal total system time. 

TYPICAL TASK. On the last entry of last Task 
Summary Sheet used on each tyre of task, there should be an 
entry for a nonexisting program. This entry should be 
weishted average (weighted by used time per month) for all 
00010015 entries for this type of task, and it should depict 
what a typical task of this type would look like. 

d. Selection of Representative Tesks 

From each of the sets of tasks, select tasks 
(preferably a single task program) which are representative 
cf the set, or a substantial portion thereof, and identify 
these tasks with asterisks. The types and time of proces- 
Sing, amount of internal storage used, language used, and 
۴۱۲۱۱ ۳۵۲ 2 configuration should all be taken into account when 
selecting the representative task; that is, it should be as 


nearly similar to the typical task as possible. 
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ES ECT (۱۱0 Factors 

A chart should now be prepared showing each of 
the representative tasks and the functions each presents. The 
Monthly times required for each of these functions within a 
task should be listed alongside of the individual times of 
the benchmarks chosen to represent these task functions. The 
individual benchmark times for the functions should be di- 
ከ. ው! into monthly times for these functions to obtain indi- 
lI extension factors, which show how many times a month 
the representative benchmark would have to be run to make up 
tne full monthly workload for the task. aAn example is shown 
in Table 10. If only sequential systems were to be consid- 
ered, the individual functional extension factors would be 
sufficient, and each vendor could run the benehmark program, 
extend his system running time for each benehmark by the 
functional extension factors just derived, and thus be able 
to tell how long his system would take to complete the total 
workload. 

In third-generation systems where many degrees of 
simultaneity must be considered, it is possible that while 
۳5 system is handling one function, it could simultanecusly 
be handling another, or even be multiply handling various 
tasks on programs. Thus, all the representative programs 
must be considered together to form the representative sample 
of the total workload. If the normal workload could be 
processed in variable ways, then a vendor should be permitted 


to handle the grouping or mix of representative benchmark 
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Mask Set 


port 
8-13 


Update 
D-53 


Matrix 
Inversion 
RÓS 


FORZRAN 
Compile 
H-3a 


COBOL 
Compile 
G-2 


Tape to 
Print 
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REPRESENTA TIVE PROGRAMS 


Workload 
Lec OS 


cal wie ne ከ በ 
Mag. tape 
Card reader 


Poca! ما ما‎ 
Mag. tape 
Card reader 
Printer 


Motel campo 
Mag. tape 
Card reader 
Printer 


Dora) chruput 
Card reader 
Mag. drum 
Printer 


Mota lathruput 
Mag. tape 
Cara reader 
EN Ven 


16 0 تا‎ 
Mag. tape 
Card reader 
Printer 


Mortal tirarse 
Mag. tape 
۶710567 


Time (hours) 


Monthly Representa- EC On 
Task tive Task Pactor 
(single run) 
145.00 0005 322 
125.00 0,25 500 
115.00 0.03 3833 
120280 و‎ 160 
COG CE 133 
200 O50 LO 
1 ان‎ OO 0:25 400 
100.090 ORIG 625 
TOROQ 858 T OO 
CONDE 60 5 00 
ONE IO 70O 
90.00 C245 200 
230 002 TE 
24.00 O15 9 
0ل‎ G O TSO 
85.00 کات‎ 500 
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300 0.04 5 
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300.00 1O0 300 
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Joslin 1977 
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programs in whichever way his system can best handle them. 
jmeenormally one type of workload would be handled before the 
other, then the mix should be structured in that way.  How- 
ever, just taking one each of the representative benchmark 
programs does not make a representative mix because the 
related extension factors also have to be considered. Table 
11 shovs a mix of representative benchmark programs and the 
mix extension factor. 

foes Of Tasks 

The extension factor for the mix is derived by 
Peomining the information contained in Table 10 and obtain- 
ing the lowest practical extension factor to reduce the 
number of problems to be run in the mix while retaining the 
required representative nature of the mix of problems, which 
in this case is 160. This extension factor is then divided 
into each of the seguential extension factors to obtain the 
EEnantlty column. The previous column is then used to make 
the input/output total time when extended by the mix exten- 
Meeretactor equal to the total projected input/output time. 
This mix of tasks can then be used as a proper demonstration 
Siena Supplier's multiprogramming or multiprocessing [Joslin 
1977, Stimler 1974]. 
2. Expected Workload Levels 

The workload to be processed by a system car be 
expected to increase over time, Therefore, the workload for 
a system can be envisioned as consisting of a series of 


various levels. These workload levels can be roughly 
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B-1a 
E-Ha 


D-5a 


K-Ca 
H-38 
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TABLE 11 


BENCHMARK PROGRAMS 


GUANTITY 


Extension 


2 


l 


(ኒን 


2 


38067 ToT 


100 


Max: 


160 


FROVESTONS 
Normal 
inoue seem tave 
Twice normal 
ducc mo, ٣ك‎ 
Normal 
Normal 
Normal 
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EEcroximated by the average monthly workload levels for each 
Ban of system life. The workload increases from one level 
to another as tbe system ages increase. Because of the 
uncertainties that are sssociated with projecting workload 
Growth overtime, it is impossible tc predict with complete 
Mmerocuracy just when the workload will reach a given level. 
Therefore, probabilities are used for this purpose as 
described below: 
a. system Life 
A chart showing system life should be prepared 
showing the number of years that the system is expected tc 
erin existence (see Figure 7-a). 
b. Prolected Growth 
À best-guess approximatior of projected growth 
of the system should ther be superimposed on the foregoing 
chart, the vertical axis depicting the workload in hours-per- 
month. Figure 7-b shows this. 
c. Workload Levels 
۶۰۰۰٠٠٠۰٠٠٢ ٣٦۷۰٢ ٣٣٣٢٠٢٠ ٦٣٦٦٠٦۳۰٠٠٠٣٦ ت٦6‎ growth line for 
each year, construct a workload level line parallel to the 
horizontal axis (see Figure 7-c). 
d. Level Probability 
For each year of system life enter the probability 
of the average worklcad for that year being at or near each 
of these levels. The probabilities must be thought of as 
lumped at these levels in such a way that the total probabil- 


ity for a year adds up to 100%, because the number of levels 
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used is finite [Joslin 1977, Svobodova 1976]. For example, 
referring to Figure 7-d, it can be seen that the workload 
there illustrated has a probability of 90$ of being at level 
one for che’ 7171 twelve months and a 5% chance of still 
being there for the secord twelve months, Therefore, each 
vendor would be asked to determine the configuration 
necessary to process workload level one in the aliotted time 


ጠመው. . 2:10 to determine the montaly eost of that configuration. 
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Level: 


M U E WU) كن‎ 


Year: 
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EXAMPLES OF LEVEL 


PROBABILITIES 
0 0 0 0 
0 0 0 5% 
0 0 5% 15% 
0 0 10% 75% 
0 10% 80% 5% 
10% 85% 5% O 
90% 5% 0 0 


lst 2nd 3rd 4th 
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FE OPSEORFEROCHRENGTCOMPUTER SYSTEMS 


The three most commoniy used computer procurement plans 
offered by the vendors are lease, purchase, and lease with 
Mpio; to purchase. The decision on the selection of 
computer hardware (and software) and procurement methodology 
is a management responsibility, and should be based upon a 
feasibility study and subsequent evaluation process. in 
connecticn with hardware selection, the manager always makes 
second decision; that is the decision as to whether to 
purchase the computer or to rent it. The fessibility study 
should include recommendations on the purchase-rent decisions 


and the facts upon which these recommendations were based. 


A. COMMON PROCUREMENT PLANS 

ime ots section the common methods of procuring computer 
systems will be introduced. 

۲ Leasing 

Leasing, in the context of computer use, usually 

means an operating lease, with cwnership of the computer 
system retained by the vendor. The user pays a predetermined 
monthly price for tbe use of a certain length of time on the 
Bempuber System, The lease price includes rental of the equip- 
ment, a fee for the maintenance and service of the equipment, 
and a payment to compensate the vendor for the risk of owner- 


ED. The length of the lease is very important in its effect 
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on the rate. Since the leases which are proposed by the 
meamueracturers are usually short-term ones, the following 
eeescussion will concern itself with these short-term leases. 

Under a normal short-term operating lease, the user 
enjoys the following advantages: 

1. Frees working capital for more productive use (since 
money is not tied up in low-yielding fixed assets). 

2. May cost less than other methods of acquiring equipment. 

May increase the firm's ability to acquire funds. 

۳۰٠ ٣٦۲٦٦.65 only a restricted (not a general) obligation 
BEEN: the company which may be satisfied by payment of one 
ENS rent in bankruptcy or three years' rent in reorgani- 
ھ‎ ۰ 
1 5. Does not appear as a liability on the leasee's balance 
Beet. 

v 6, Leaves normal lines of bank credit undisturbed., 

7. Permits 100% financing (as against 75% or 80% through 
other methcds). 

9. Creates an allowable cost (or acceptable cost according 
O the government regulations including interest cost) under 
EEvernmert contracts. 

9. Permits hedging of business risk (primarily the risk 
of obsolescence). 

10. Minimizes danger of being oversold. 
ll. Assures more adequate servicing (since maintenance is 
Bae responsibility of the lessor and usually included in the 


lease contract). 
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^ 12. Offers the convenience of making only one payment 
(rather than separate payments for debt service, maintenance 
Bost, insurance, property taxes, etc.). 
13. May be tailored to the leasee's computer system needs 
more easily than ordinary financing. 
14, Avoids the necessity of selling equipment no longer 
wanted. 
15. Permits middle-management executives to acquire new 
۳ ۱۱ ent without going through formal appropriation request 
procedures. 
16. Provides cost-cutting equipment to be installed 
immediately. 
17. Acts as a hedge against inflation. 
/ 18, Provides long-term financing withcut diluting owner- 
ship mee control. 
From the point of view of the leasee, equipment leasing has 
the following disadvantages: 
M oui omenat Leasing charges a higher interest rate (than 
the leasee's regular interest rate). 
2. May provide less attractive tax deductions (than inter- 
est plus accelerated depreciation). 
3. Gives any residual value of the equipment to the lessor, 
4, Establishes a fixed obligation against the company. 
2. Does not provide whatever prestige that goes along with 
Ownership. 
6. Raises the fear of dispossession if payments are not 


made during hard times. 
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The importance of the listed advantages depends on 
۲ individual company and the environment in wnich the com- 
Ber is to be used. System obsolescence within the organi- 
EN n That is using the computer has troved much more 
significant than technological obsolescence. The knowledgeable 
Emos pective user, in forecasting the life expectancy of the 
BroposeG system, should carefully study ana evaluate his data- 
processing needs. However, since every user does not exercise 
the same degree of foresight in planning, the vendor must set 
his lease charges so that they allow for the average system 
life expectancy. This is a compromise measure since the 
vencor must deal both with those users who plan and D wno 
do not. One can therefore see that if the user has done a 
Eos job of planning his systems, he is in a better position 
5 ۱ ۵۱1۵ any risk of system obsolescence than the manufa:- 
Seer is. Also, it can be costly for the user to lean on 
meeoLrescence as a crutch or to allow it to influence the 
lease /purchase decision. 

Another important disadvantage of leasing is that on 
a leased computer system, extra usage is more expensive than 
it is on an owned system. And if the user makes any serious 
attempt to make the computer pay for itself, he may have to 
utilize these extra shift hours [Joslin 1977, Vancil 1962, 
5683 ٥ 1973]. 

Ihe concept of the third-party operating lease on 
computer equipment originated in the United States. Indeed, 


even in Europe, the service has been provided almost 
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exclusively by US-based companies. Its origin can be traced 
as far back as 1961 Woo. Boole Te. wrote an operating 
lease on an IBM 7094 for Ling-Temco-Vousht. The principle 
attraction to the user was a saving in additional use cnarges, 
then 40% of the primary shift rental. 

Second-generation computers were not reslly amenable 
to leasing because any significant increase in power or 
capacity required a change of hardware, so tnat even a 
medium-term commitment would not have been tolerable. “his 
constraint disappeared with third-generation systems which 
permit ችንን ን ን ር. throush addition rather than 
change. At the seme time, the total cost of running a com- 
puter was steadily mounting; because of the increased power 
wn Sophistication of the equipment, correspondingly more had 
Se spent on software and other supporting functions, The 
overall cost meant that the user had to think in terms of a 
longer life span for each system he installed, in the range 
of say three to five years [Gusta fson 1973, Szatrowski 1976) . 

The hardware itself is now sufficiently reliable, 
Bex 2 Ole and modular for a functional life span of at least 
ten years to be foreseen, against four years for rental to 
become equivalent to the purchase price. Being prepared to 
wait longer than this to recover their costs, the leasing 
companies could previde the equipment at less than the manu- 
facturer's rental. 

The subject of computer leasing revolves almost 


entirely around IBM computers. Because of IBM's large market 
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share, early in the game the computer lessors elected to 

Nm ር: 1101 equipment since they felt it had the best chance 
of being placed elsewhere. A company with a small market 
mare would create unacceptable risks to the lessor. The 
initial success of computer lessors so far as their ability 
to raise large quantities of capital and to convince many 
users of the viability of the concept is well known. But 
anses in IBM policy curtailed their growth. 

For many years IBM was a one-price shop. That is to 
say, it made no difference whether you used one computer or 
ten, your unit price was the same. It made no difference 
whether you could use the equipment for five years, ten years, 
or one year. Your price was the same. It made ro difference 
et technical requirements that you could project did not 
indicate or consider Sieni eonen Rowin. Accordingly, many 
users were subsidizing the requirements oi the more sophisti- 
cated user, Computer leasing then became a viable alternative 
because the computer lesscrs were able to offer leasing pro- 
grams that more closely matched a customer's requirements. 

The discount offered by the computer lessors is the 
obvious advantage to the user as compared with leasing from 
the manufacturer, Services provided by computer lessors vary 
substantially from company to company. Some provide services 
on the theory that they will enhance the ability to move 
equipment around. Others got into other services for diver- 
sification reasons with the expectation that they would be 


getting into other profitable businesses. For example, at 
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one time Randolph Computer Corporation owned a number of data 
Meecessing centers in the Pacific Northwest and in the Mid- 
west. These provided a whole range of services normally 
mend in such centers but really had no direct relationship 
۲۰ che computer leasing activity. It gave a reservoir of 
ENS in programming and systems engineering that could be 
used from time to time in the leasing activity. However, 
only very rarely were these skills found to be required. 

. የ. የበር oropressec, computer Jessors have recognized 
ከሺ ስክ they cannot look at themselves exclusively as offering 
EE panolal service, that is to say, renting a computer at 
a price that is less than the user would pay IBM. The busi- 
ness has become increasingly technical in nature. It is 
BeviousSly beneficial if it can be demonstrated to a user that 
there are more effective equipment configurations to meet his 
Beguirements,. This is advice that he will not always get 
from the manufacturer since it sometimes implies less 
equipment [Yearsley 197/3; GUS SON (5, Randolph 1974]. 

it can be told to the customer how efficiently his 
equipment is being used by the use of a hardware monitor. 
Ihen the customers must frequently be assisted in upgrading 
from one model to another. For example, upgrading from a 360 
model 20 to a model 30 has many software ramifications and 
m Tres a considerable amount of handholdinz. 

There are some risks to a company doing business with 
mecomputer lessor, As with maintenance, in the early days of 


computer leasing there was some fear on the part of the user 
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E cutiing the umbilical cord to IBM, he might be cut- 
۳۰۰۰۰۰ ٦5ء٥۶‎ ۵021 from valuable services. It is alleged that 
there have been instances where the salesman on an account 
۲۳ have implied that this was so. This was basically 
ሽን ሞጋ to the ground rules under which IBM is supposed to 
Moperate. Fortunately, these instances have not been fre- 
quent. The more important risks lie in the area of whom one 
Besse to do business with. Some of the computer lessons 
have gotten themselves into financial difficulty as a result of 
unwise diversification efforts. Others have withdrawn from 
the computer leasing field because they find that their 
diversification efforts have been so successful that they 
E cvred CO concentrate on them rather than on computer 
leasing. Customers doing business with such companies 
8811 2=1፣ና do so at their peril. 

simon tant Chat a computen lessor be chosen- for 
۱ financial stabilivy and its demonstrated abilicy to stay 
in the business for the long haul. Flexibility that customers 
require can be met only by someone who is whoily committed 
to this business [Gustafson ۱۳0 یر‎ 1972] 1 

It is also important that the equipment be mairtained 
The best condition. Routine maintenance is, of course, an 
۳۰۰ ۰۹٥٠٢ part of accomplishing this. In addition, when 
Eupment is moved from one customer to another, it frequently 
goes through a refurbishing center where catch-up maintenance 
is performed. The pressures of day-to-day work at an instal- 


lation often will not permit 411 the maintenance routines to 
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۲ been effectively completed. This can be accomplished at 
a refurbishing center, It is also important tnat installation 
be performed smoothly and with a minimum of disruption. Proper 
Eunstallation planning is an essential ingredient. Cne of 
the measures of the effectiveness of any computer leasing 
organization is how well it contends with emergency situa- 
tions as they occur [Randolph 1974, Oliver 1973]. 
a. Lessee Motivation 

However mixed the options of the manufacturers, 
however varied the fortunes of the leasing companies, the 
computer user who employed a leasing facility nas generally 
End it highly rewarding. Without any significant change in 
his relationship with the supplier, the user has been abie to 
obtain exactly the same equipment for between 10 and 25% less 
than the normal rental charge [Graham EE 

Substantial cost savings have, therefore, been 
memes principle benefit to the user. Also, leasing offers 
emouner option in the choice of acquisition method. Tradi- 
tionally the manufacturer offered two alternatives: rental, 
Ey for a minimum of twelve months only, or outright 
purchase. Leasing provides a further choice: a lower 
rental charse for a longer period of commitment. The 
previous rental user and the previous purchase user both 
ا‎ che leasing proposition attractive. This is because 
the conventional alternatives are best only in extreme situa- 


tions which are not the normal user requirements: short-term 
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rental is suited to the user who wishes to make frequent 
major changes to his equipment, and purchase to the very 
stable environment where a commitment of six years or more 
is acceptable and where, also, capital and credit are readily 
available and not better employed for other purposes. 

In practice, even for the rental user, major 
equipment changes cannot be economically made in less than 
two to three years so that the flexibility provided with a 
twelve-month agreement is more illusory than real; it is in 
fact a relic of the punched card era when change was less 
መክ 1 ህን 18. 105 effect on a company's overall activities. 

The purchase user already realised that the 
manufacturer's rental terms offered flexibility he did not 
need at a price he did not want to pay. However, the purchase 
esena tive contained two deterrents: first, commitment to 
hardware over a period long into the future (at least six 
years) during which unforeseen requirements could arise for 
wemeuber processing power, and during which technological 
advance could make the equipment obsolete; and second, tying 
Bp large amounts of capital or credit which could normally 
be employed better elsewhere in the company, in its own line 
of business. 

Leasing, therefore, provided a very acceptable 
compromise between the extremes of rental and purchase, and 
the commitment of two to five years, tailored to the user's 
plans, was less of a hardship than 2 correlation with real 


requirements, Furthermore, different components of the 


12175 





system could be rented cr purchased if these methods of 
acquisition were selectively found most suitable. For example, 
NS faster printer is planned one year after tne main instal- 
lon, this could be rented direct from the manufacturer, 
while the rest of the system is leased. 

In addition to the saving against the manufac- 
turer's normal rental charge, leasing contains a further 
tangible advantage. The leasing company's charge normally 
covers use of the equipment 24 hours a d3y, whereas the 
manufacturer's standard rental is for a specified number of 
hours per month, roughly equivalent to a single shift five 
days a week. An additional charge is made for use beyond 
Maes period. It is true that the leasing customer will 
w ably incur an additional charge for maintenance, but 
this again is less than the rental alternative [Graham 197%, 
Coutinho 1977, Tatham 1969]. 

There are a variety of ways in which the leasing 
customer can capitalize on the benefits available, Most 
۲۰.517, 026 can reduce the cost of his computer installa- 
tion. Alternatively, he can have more equipment or more 
people for the same expenditure as previously budgeted under 
the manufacturer's rental plan, In addition, there are more 
far-reaching opportunities for the user, the benefits of which 
Eula iar outweigh the direct savings. Through leasing, it 
may well be possible to install a system of greater power than 
Originally envisaged, and then keep it for longer. A leased 


1500/40, for example, may cost as little as a 360/30 in about 
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two years, The user who needs the power of a Model 30 now 
and a Model 40 in, say, two years' time could afford to 
install the Model 40 at the outset and then hold it for five 
years [Yearsley 1973). 

The real advantage from doing this is not only 
"in acquiring more power for the same money, but rather in 
conferring stability on the data processing departments. Dy 
avoiding frequent changes in hardware the user avoids the 
concomitant expense of software and procedural changes. 
Maeres there are frequent major changes in equipment, the 
ME Tes and costs of the data processing department are 
expended on technical transition from one language to another 
or from one operating system +0 another. This does noct make 
"ን for the company. However, by first establishing a 
stable technical environment for, say, five years, the data 
processing staff can then concentrate cn its main purpose of 
miereas. ng the computer's functional contribution to the 
company's business. In this way. the leasing facility pro- 
frees the opportunity not only for better value from expen- 
EM Ure on the hardware itself, but for better value from the 
whole computer investment. 

Major companies tend to predominate among the 
customers of the leasing companies for a number of reasons: 
they have more to gain in absolute terms, they were quick to 
perceive the advantages, and they were attracted to the 
leasing companies because of their credit standing and 
because they tended to have medium to large computer systems 


which most iessees favored [Randolph 1976, Graham 1973] : 
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5٣ Motivation 

Like che lessee, the lessor is in the business 
for the money he can make out of it. Already a number of 
entrepreneurial fortunes have been made (and some lost) ir 
the USA from this business and some companies of substance 
have emerged, already diversified into other fields. 

The lessor's starting point is his willingness to 
take an eight- to ten-year view of the computer as a revenue- 
earning investment. He supports GER Dosador inda niber. of 
۲ first, the computer is an electronic device with Little 
to wear out; second, third-generation computers are suffi- 
clently reliable and modular to have a long working life; 
third, twney have proven to be compatible with the next 
generation of nardware; fourth, the pace of technical change 
۱ che computer industry is slowing down. 

The lessor's view of the machire is thus quite 
es erent from the user's, It is also quite different from 
Eus of the manufacturer's. In developing and building a 
computer, or family of computers, the manufacturer has 
invested huge sums of money. Even before the first computer 
of a new 'generation' reaches its first customer the manu- 
facturer has spent millions of dollars on research and 
development, and on plant and equipment. He has to recoup 
this within as short a period as the market place will allow, 
in a market place which is rental-oriented. In practice the 
selling price of a computer is normally recovered in approxi- 


mately four years of rental [Bucci ትን EU US 1975]. 
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So the Manufacturer's rental-to-purchase ratio is 
four years, the user thinks in terms of three to five years, 
and the lessor believes it will earn reasonable revenues for 
eight to ten years. Here then lay the opportunity for pro- 
መክ ከሮ an attractive service which could itself become a 
substantial business. Leasing gained rapid acceptance and 
generated large and fast-growing profits for the lessor, 
These were to some extent dependent upon the rate of 
depreciation, and true profits would be obtained only when 
the lessor had fully recovered all expenses at some time in 
the future, based on the ability to remarket equipment when 
the first user had finished with it. This remarketing capa- 
bility certainly did not exist, nor was it needed, when the 
initial leases were being written. The lessor would certainly 
need this capability and/or other business activities to 
offset the risk inherent in the leasing operations themselves. 

Ihe leasing companies did not have to wait long 
to satisfy these requirements. The size of profit they were 
generating and their rate of growth rapidly captured the 
ES mation Of the investing public in the USA, providing a 
weed multiple for the company's stocks. This in turn gave 
Bae leasing companies the opportunity for acquisition and 


diversification [Yearsley 1975, Oliver 1973]. 
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c. Lease Contract 
ከህ لا وی لو اه را دا‎ characteristics of 
meen the purchase and rental contracts. In the computer 
industry, lease contracts are available through "third- 
parties", or directly from the vendors. The third-party 
company will purchase the equipment from tne manufacturer and 
lease it to the user. The terms can be flexible and negoti- 
able, depending on the risk to the lessor; thus the longer 
the duration of the lease, the more favorable the terms and 
conditions possible to the user. The lessor must rely on the 
Bash inflow (depreciation tax deduction plus cash payments) 
and the residual value of the equipment to cover his costs. 
If the term of the agreement is of relatively short duration, 
Mier Tessor must look forward to the problem of finding a 
second user, 
Tease contracts fall inte two general categories: 
l. Full payout or financial leases. 
2, Non-full vayout or operating leases. 
In the full payout or financial lease, the user 
(or lessee) essentially has the rights of purchase and 
assumes the risks normally assumed by the purchaser. The 
legal title, however, is retained by the lessor. The lessee's 
payments are designed to recover for the lessor: 
1. The total cost of the equipment. 
2. The cost of money required to purchase the equipment 


by the lessor. 


3. A contract fee, normally atout 0.5% or more. 
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reina tion.) cae lessor still owns the eguic- 
ment, although the lessee will normally have the option to 
ese. The full payout lease is normally used to obtain 
financial benefit for the lessee; for example, lower taymerts 
myer the useful life of the computer as compared to a rental 
[Szatrowski 01۳۰۰-۰۹۹۰۰ و3‎ ۶ Bos. Bucci 1973]. 
The non-full payout or the operating lease has 
Ra; characteristics of a rental contract. The essential 
difference is the length of commitment. The term of this 
contract generally starts with a minimum commitment of two 
years, and can go as high as ten. Monthly payments average 
10% to 30% less than the manufacturer's rental price. 
Illy Speaking, a lease contract (either financial or 
E ting) is the most flexibie of all contracts abailable 
to a user of computer equipment, The user can negotiate with 
the lessor for terms most beneficial to both parties. These 
negotiations are somewhat unusual since both parties, by and 
large, are aware of each other's financial needs and require- 
ments. Some items that affect the negotiaticns are: 
l. Maintenance 
2. Depreciation 
3. Investment tax credit 
4. Property taxes and insurance. 
One of the two ade must pay for maintenance, 
and the cost is the same. for either party. There may be 
local advantages for one party or the other to assume the 


malntenance obligation. For example, the user may already 
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۲۳ ۰ a maintenance contract with the manufacturer for other 
EEuputer equipment and could perhaps extend it to include the 
leased equipment. Alternatively, the lessor may have a 
Sonal contract with the maintenance organization. 

The investment tax credit is a direct tax benefit 
E one of the parties. In certain cases, it could benefit 
one corporation more than another. For example, if one of 
the companies may be operating in a loss period, it may not 
need the investment tax credit since its tax would rot be as 
meee as in other periods. Another case might occur when a 
company makes massive investments, say an airline in the 
years it purchases new planes; such investments may exhaust 
RS potertial investment tax credits. In such cases, by 
relinquishing the investment tax credit, the user may be able 
to negotiate a lower lease price. 

There ere some additional tax considerations to 
be taken into account ina leasing arrangement. For a trans- 
action to be acceptable as a true lease, i.e., not as an 
m ۵ 11۳۵۲ سا‎ purchase contract, the lessor is required to 
me à significant risk both during the lease term and in 
eeepc riod aiver its expiration. According to IRS reguia- 
tions the ideal lease arrangement would have characteristics 
among which are these: 

l. Lease payments would te approximately the same through- 
out the basic lease term. 
۳ |. options are nct at fixed amounts but are based 


۳ ۱23135 values at the end of the lease term, 
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8. The estimated fair market vaiue of an asset at tne end 


et's original 
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Sf the lease term is at least 10% of the as 
ROS Lt. 

The lease term is less than 20% cí tne asset's useful‏ ہلا 
Inte.‏ 

٠۰۰۰۰٠٠ ہ1‎ 363۰٠٠٦٦67 8813 17818 ٤ tas 
ہد‎ eraot ካር 151101 5604 25115351112950 01 the equipment 
for the lessee, There are also no overtime use payments 
Associated witn a lease ccntract [Szatrowski NOME OSE Boe 
1973, Sabcl 1972, Randolph 1974). 

۰ ۶2۱170٥۹6 

When the user purchases the computer system, he 
acquires ownership of it and can use it ore shift or around 
tne clock, seven days a week, with minimal increase in hard- 
ware expenses. Maintenance and service cr the computer are 
contracted for separately with tne vendor. With respect to 
taxes, the user may depreciate the purchased computer system 
ES ne would any other item of capital equipment. Any opera- 
tion of the system beyond the break-even point constitutes 
pure crefit to the user-owner, for he avoids those lease 
payments which he wculd be making had he decided on an oper- 
ating lease [Randolph jou ucc OM Graham 1973]. 

ihe break-even point can be calculated as follows: 
The number of months to break even equals the purchase price 
divided by the difference between the monthly lease cost and 


the monthly maintenance cost; (see Figure 3) 
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BE, = —— — 
m 
DOS - o 
E = the number of months to break even 
Po = the purcnase price 


LC, = the monthly lease cost 
Mm = the monthly maintenance cost 
The user stands to enjoy certain tax benefits, but 
he assumes the normal risks associated with ownership: If 
۲۲ system fails, che responsibility is his and not the 
manufacturer's. When he considers purchasing, the user ougnt 
to take the time value of money into account: loney Spent 
today is more costly than the same amount of money spent 
some years from now. Given ar interest rate of 10$ rer annum, 
۲ ۱ 16۳5 dollars used to purchase a computer Today has the 
same value as 1.6 million dollars spent five years from now, 
a. Purchase Contract 
Under a purchase contract, tne purchaser bears 
۳۳ ۳6 risks of ownership including insurance, taxes, and 
equipment obsolescence. 
By and large, the purchaser will obtain the seme 

Services and support from the vendor that are available under 
e lease or rental agreement. There are, however, three 
Important factors affecting this financial decision: 

l. Full payment must be made to the vendor upon delivery 


of the equipment. 
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2. A separate maintenance service contract must be nesc- 
mrated Since service of the equipment is not considered part 
መሽ ር: purchase price. 

3. Insurance premiums and appropriate taxes must be paid 
on the asset. 

Assigned values of depreciation can substantially 
affect the cash flow analysis for a purchased system. The 
buyer of any expensive capital equipment should be acquainted 
with the optimum depreciation schedules allowed by law. In 
cion, the future projected tax position in the corpora- 
mor should be considered in order to be able to calculate 
Mes after-tax cash flow. 

D Tha assignment of a residual (or market) value 
to the equipment at some future date is probably the most 
difficult estimate to make in the financial analysis.” If the 
residual value is too optimistic, losses are experienced at 
resale or trade-in time. On the other hand, assigning a 
zero dollar value as residual may be entirely unrealistic. 
Under such circumstances, it may be advisable to assign both 
we most pessimistic and the mest optimistic value for 
ual, with analysis urder both conditions. Statistically 
it may be possible to determine the most probable outcome 
under these circumstances [Szatrowski OO Joslin LOTT, 
Randolph 1974] . 

b. Rental Contract 
Under the rental agreement, the user is liable 


for a prepaid fixed minimum payment. The agreement can be 
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۰۰۰309٦668 by a minimum of 90 days prior written notice. 
Under this agreement, the risk of ownership remains with the 
vendor. The user mas memobi)estiems:ror such expenses as 
masurance and maintenance; however, he is responsible for 
Em taxes that might be levied on the rental contract by 
Me state or local government. 

Extra shift use, over and above the standard 
memonty base hours, represents an additioral cost to the 
sS Investment tax credit is also a consideration under 
a rental contract and can be passed to the user. Rental 
መጠው ጋርኒርሮ find a high level of usage in the computer industry 
۲ co a number of factors: 


I Low risk 


2. Financial leverage 
۳ equipment obsolescence 
۱ Flexibility 


የ ٦3.٥6.7۱36 propia) tne best argument- for a 
Memcal contract. when the user has a ccntinually varying 
RE oí jobs that requlre different configurations of equip- 
ment, it is to his advantage to be able to move equipment 
uly in or out of the installation without penalty charges. 
Straight purchase has two serious drawbacks: 
l. It reguires a relatively large sum of money all at one 
period of time. 
۲ Lt dees not permit the activity to adequately test the 
equipment? and their system before they have committed them- 
selves to buying something which may turn out to be too large 


er too small. 


NET. 





Suda. purchase plan should be used only when 
۳۳ system life, including reuse, is longer than five years 
and the equipment has had emple time to demonstrate its 
TT Lo handle a proven application and purchase money is 
available [ Szatrowski More, Coutinho 1977, Gustafson 1973] . 
Meloy, a revolutiomsvas taking place in the 
prices of medium and large-scale computer systems. Prices 
Rom hardware fell dramatically. Table 12 gives an idea about 
these prices. 
Er pleasing with Purchase Option 

The main advantage of the lease with purchase option 
EN US: there is a trial vceriod in which the manufacturer's 
system is tested in the user's applications. If the system 
Annot satisfy the reguirements of the applications, it can 
be replaced at relatively little expense to the user. If, 
however, the system fully satisfies the requirements of the 
EDU caclions, the user can carry out the purchase by exer- 
Eine che opticn and buying the system. In this case, little 
money will nave been wasted since the larger part of the 
total lease payments can be applied to the purchase price. 

sometimes the purchase option has tc be negotiated 
Separately with the manufacturer, and sometimes. it is a 
E Sndard part of the lease contract. The intent cf this 
۲۰۰۰90٦٦15 to give a user the right to purchase the system 
Within some speciried period of time, normally one or two 
Eu. iJ: the right is exercised, some stated Bersenvaze of 


* 


the money paid to the vendor is applied toward the purchase 
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Brice. The option removes a great deal of the risk involved 
w nership of an untested system. The user has a chance 
EN Cc his applications successfully run before he agrees to 
purchase the system. Lease with purchase option plan should 
memiised if funds are available and if the system life, 
including reuse, is longer than five years, and if it is more 
Beenomical and practical than the other ownership methods 
٣۰٠۰٦ 0 ۰ 
4, Lease to Ownershio 

Mais plan Is new on the computer procurement scene 
ard is knovn by several different names: Special Lease- 
Purchase Plan, Alternate Payment Plan, Installment Purchasing, 
etc. These plans are all essentially the same; in thet, 


monthly lease payments ere made until some given number 


(1 


(generally sixty payments) nave beer made, or until some 

٣٠٠۰ amount (the purchase price of the system) has been paid, 
E hen title of the ccmputer passes to tie lessee, Until 
that time the lessee has no obligation beyond a normal lease 
plan [Joslin ا‎ 1973]. 

Lease to ownership plans are not offered by all tne 
vendors since they are new. Rather, they are made available 
only upon request. Late in 1969, the Automatic Data Processing 
Equipment Selection Office (ADPESO) of the Department of the 
Navy started requesting that some form of a Lease to Ownershio 
Plan be offered as one of the procurement alternatives in 
۲۳ ۵9 15 submitted in response to the request for prorosals 


meeued by that office. After much ignoring of that request, 
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one vendor finaliy offered a Lease to Ownership Plan in 
Peace of offering a discount on the cost of his system. 
Miter that, several vendors offered similar plans and have 


Mencinued to offer such plans on subsequent bids | 65138 
1977]. 

In order to make Lease to Ownership Plans an accept- 
able procurement alternative, three major problems nave to 
men overcome: 

1. The evaluation technique used in selecting the computer 
system must be broadened to consider value outside the stated 
system life, 

2. The purchase alternative of procurement must be 
re-examined and reconsidered on more tnan just a cost basis, 

3. The tax situation relating to ‘gradual ownershir' of a 
capital investment must be investigated. 

companys princ plefinterest $n Lease to Ownership 
prans might be due to their recognition of the advantages of 
ownership, coupled to an understanding of the unavailability 
ENDbBrohase funds within their present economic situation. 
If purchase funds are not available, then some form of Lease 
to Ownership Plan is essential if the activity ever wishes 


to own the system. 
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B. LEASING VERSUS PURCHASING 

Ene case oi mos; pusiness problems, che purchase-‏ ا 
or-lease problem should be settled on the basis oí a careful‏ 
Bonsideration of many factors; both quantitative and quali-‏ 
tative, These considerations must be balanced against one‏ 
another to come out with the optimal solution. One method‏ 
co consider the relationship between the cost or purchasing‏ ۳5 
and the cost of leasing.‏ 

Mae quantitative factors are: 

۳ Cost involved in purchase versus lease arrangements. 
2. The estimated useful life of the machine. 
3. The desired rate of return on investment (ROI). 

Where applicable, income tax benefits and salvage values 
should be considered. By comparing the cost of purchasing 
with the cost of leasing, the financial advantages of both 
methods may be studied.  $uantitative analysis can assist 
management in deciding which method of acquisition to use. 
Methods of quantitative analysis are described in Table 13. 
Other costs and factors, such as insurance, risks of owner- 
ship, resale prices, income taxes, and so on, are disregarded 
۳٠۰۰-٣۱٦6٦17 effects can be easily included in the analysis 
when required. 

l. Methods For Lease-or-Purchase Decision 

a. Method I 
Ihis method ccmpares the cost of purchasing with 
the cost of leasing without considering rate of return. 


Cumulative costs incurred, namely purchase cost ir the first 
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TABLE 13 


METHODS OF GUANTATiVE 


Purchase basis 


Purchase cost: 


Estima ted useful life: 


Maintenance charges: 


First three years: 


Last three years: 


lnterest on borrowed 
funds 
(desired rate of 
return) 


ANALYSIS 


$600, 000 


6 years 


545,000 
per year 


$55,000 
per year 


io 


Lease basis 


Rental charges for 

irst Six years 

(including 

maintenance 

charses) for the 

same usage as 

would be the 

case if 

curchased 

SDR وا‎ 0022020 
per year 


mE 








Rear and maintenance costs in the first year, second year, 
and so on are used. .For the example in Table 14, the break- 
Ben point would occur just before the fifth year of usage. 
Thereafter, the purchase basis affords favorable financial 
advantages. 

b. Method ii 

Ina srapproschseonsidenss purchase price, mainte- 
nance charges, and rental charges amcrtized over the useful 
۱۳ oí the equipment (including interest costs) in equal 
amounts. This approach also, as with Metnod I, looks forward 
Precumulating technique) in analyzing the effects of pur- 
chasing outright or leasing. However, there are tuo basic 
differences: 

l. The purchase price, maintenance charges, and rental 
charges amortize in equal emounts in time, 

meiner interest lor rate of return) applies to the unamor- 
tized amounts during the period, 

Both of these factors could be assumed to occur 
monthly or more cften, but here the amortization and interest 
computations are assumed to occur annually. Since the rate 
Of return would be computed on the book value of the invest- 
ment, the interest (rate of return) for each year on the 
purchase price would be as follows: 


First year: 


$600,000 x 3$ ......... $18,000 
SOC HCCC x BD .. mM... MME ,000 
100,000 x BO. ... NE SOD 
300,000 4 9,000 
از امس‎ .:5.... 1 6,000 
LOO OOO E SD Lic. 3,000 
Rotal O -::: $63,000 








T OB ۵ 
E mE mE 5 


A ےہ‎ 


ادا وکا 731۱ 
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03 1 100115 fOreune above Computation can be 
expressed as B+ Í (where n is the number of years), and 
2 
can be derived using the formula for the sum of an arithmetic 


Brosression and the series, expressed as follows: 








E በ. በ650 2 TT 1 - (n =l) . n +1 
S = n/n + ==> s š ae 
ን, ቲዲቲፕቲያ፡. e csi Lhe Lormula tor m; 
the interest can be computed as follows: 


e+ 1 
E 





00 0055600000009 128x895 m50900 = $63,000. 


Similarly; a formula can be derived for computing 
interest on the maintenance charges and the rental charges; 
however, in each of these cases, the annual payments are 
considered individual investments when payment is made. 
Therefore, instead of one simple multiplier of un there 
are series of such amounts. The result is the following 
multiplier: 2- (2 +, l +1). 

E 2 ( 2 2 ) 
The application of the above multiplier would be as follows: 


E + E + 1) x 3% x $ annual maintenance or rental 


charge (or incremental charges 
if there should be a variation 
in annual amounts) 
As an illustration, the maintenance charges of $45,000 under 
the purchasing alternative would result in the followins 
interest (or rate of return) on the payments: 
27 ۱ 
E 89987 x $45,000 = $18,225 


This would be for six years. Since there is an increment of 


$10,000 increase beginning in the fourth year, additional 
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interest (rate of return) would be $1,350 computed as follows: 

EN 4950555530. 9002 5193560 
The comparison of the two alternatives, using the above 
EMEN CN, would result in à cGecision t5 rurcaase coe equipment. 

To lease the equipment would rean an increase in 
cash outlay of $298,425 over that of outright ownership of 
the equipment, computed in Table 15. The above computations 
assume that all of the amounts are amortized over the useful 
life of the equipment in equal amounts, except for the 
incremental increase in maintenance charges. 

es Method III 

re cthodsuses- trrezdisesunting technique 
(present value method or discounted case flow) frequently 
meee in Other financial situations. It is illustrated by the 
edule in Table 16. 

The present value method shows the effect cf 
Bmeeuding interest cost in addition to the other ccsts as 
shown in Method I. In contrast, the break-even point (indi- 
cating that the purchase method is financially advantageous) 
does not occur until after the equipment has been used for 
Eré chan four years; i.e., sometime in the fifth year of 
E Ee, If other costs and factors are considered significant 
and they can be expressed in monetary terms, Method III 
permits an easy approach to the determination of the total 
financial advantage, 

Since column (f) in Table 15 is a computation of 


the present value at the time the purchase-or-lease problem 
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Buses (at the beginning of 1973), the financial advantage 
E purchasing the equipment under the present value can be 
calculated as shown in Table i7. The present value approach 
reduces the rental charges (and maintenanoe charges) to their 
present value for direct comparison with the purchase price, 
since the purchase price is already stated in terms of its 
present value, 

Another approach 053225 Lhe present value method 
is the incremental approach which sets forth when the payout 
occurs, as shown in Table 18. 

Table 19 shows present value factors, or conver- 
Ben factors to convert future cash flows into present 
Mewes. They are factors that when multiplied by an amount 
to be paid in the future, give the present Jiscounted value 
of those funds. For example, at 6% interest, 31,000 to be 
paid in one year is equivalent today to $943.40; $1,000 to be 
paid in two years is equivalent today to $890, etc. ፲፻ 
۱9 Lead of money to be paid out, it is money to be received 
(or figured in tax deductions), then again a $48,000 tax 
savings a year from now (at 6%) is worth $48,000 x 0.9432, 
or $45,283.20 today. | To give an ides, Table 20 describes 
the IBM 370/145 ہت‎ system configuration. The purchase 
column shows the manufacturer's new purchase price for each 
unit. The prices shown are typical and will vary depending 


en optional equipment and manufacturer's price changes. 
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pres 2 ኢ. ara 24 give the examples of 
Present Value Cash Flow Analysis for manufacturer's rental 
Bemoract, third party lease (full payout), third party lease 
E 1111 payout), and purchase (with zero residual value) 
respectively (assuming an IBM 370/145 configuration at 
595131,835.00) [Gustafson 1 Szatpowski 1976, Fowler and 


Starr 1974, Randolph 1974). 
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۷۸۰۷۰۰۹ 0ط‎ TEENS EROFOSALS 


٦۶۰ت.‎ Speeification Trerarazion, thought must be given 
ED Le problem of soliciting and evaluating proposals. 
erging proposals is no real problem; the problem comes 
NN ceping control of the solicitation. The necessary con- 
tact with the vendors is usually a difficult thing to con- 
Eu One or two vendors usually have managed to become a 
EX: che "family". In fact, thoughts about the need for 
a (new) computer system probably were initiated by a vendor. 
Investigation of? the system requirements developed by the 
system study team are more likely than not to have uncovered 
several equipment requirements that were unique to the 
inside vendor's systems. It is precisely because of items 
ከክ በ 1= that vendor contact must be controlled. 

The system requirements of the company are rarely in 
accord with the system capabilities of any one vendor's 
Ester. The purpose of the acquisition is to find the com- 
Eu om which most closely iulfills the system require - 
ments (not to make the system requirements echo some given 
vendor's equipment capabilities). 

The objectives of the vendor are not the same es the 
company's objectives; since the company will have to live 
with and pay for the selection it makes, the company's objec- 
tives should be met. The only effective way of assuring that 
mae vendor's objectives are nct dominating the company's is to 


remove most of tne influence by removing the vendor. 


1,2 


| 
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At some period during the system study, vendors Known to 
Have computer systems which might be able to nandle the 
system requirements should be asked to discuss tneir systems. 
Then all vendors should be "locked out" so that they ro 
longer can influence the final system specificatiors. 7p to 
the lock-out, one vendor usually will have exerted the most 
E ence. The purpose of the presentations by several 
vendors is to reduce this influence and vo demonstrate that 
other vendors' systems have desirable features. 

The lock-out requires that one individual within the 
company be established as the sole point cf contact with all 
ቼ ከትቢያ ፡ This person should not be involved directly with 
the preparation of the system specifications. The lock-out 
Sad continue for the full period of the selection. The 
B umderwMof this section will concern itself with the 
tenor controlled solicitation and evaluation of the 
proposals. The review of the system requirements, covered 
Serer, is the basis of the solicitation, but there is more 
۳۰۰٠٠٠٠٠٦٦ 7٦ 666105 thar just supplying system ee 

The vendors could be asked to bid after being supplied 
with nothing more than the system specifications (require- 
merts), a few statements about necessary vendor support and 
mee due dates for the submission of proposals. Such a bid 
request might be sufficient, but it does not ensure effective 
Fender contact. Effective solicitation of proposals includes 


botn a good specifications package and good vendor contact. 
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A. REQUEST FOR PROPOSAL (RFP) 

A good RFP package should contain, at tne very least, 
statements concerning the following twelve elements [Josiin 
1977, Thrussell 1976]: 

1. System requirements 
57۸7511773۲61001 criteria 
System support 
Benchmark data 


Bidders' conference dates 


Cneck-in dates 


Provision for handling questions 


Proposal due dates 


VO X05 = ea ال‎ He ان‎ O 


Vendors' demonstrations and presentations 


| 
O 


Contracting conditions 


Award dates 


H- 
E 


12. General comments 

SYSTEM REQUIREMENTS. This section should contain the 
system requirements, as defined previously. Any limiting 
conditions that were uncovered during the study should also 
be included. 

ENSEDATION CRITERIA. “The criteria by which the proposals 
are to be evaluated should be explained to the vendors. This 
includes both the factors to be evaluated and their relative 
values and is beneficial to both the user and the vendor. 

The vendor gains in three important ways: 
l. By knowing the rules of the game the vendor is better 


able to decide whether he wants to participate. The decision 
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to play is an important one to the vendor, for while he 
realizes that the returns are high, he also knows tnat the 
entry fee is high. Before ne can ever nope to be awarded 

Re contract, he must prepare and submit s proposal, “o 
prepare a proposal, a vendor might incur a cost of from $500 
for a response to a simple hardware specification, to $20,000 
or $30,000 for a proposal in which detailed system study is 
required, to over $100,000 for protracted studies of a multi- 
EM ut; ofi systems whieh might additionally involve larger 
benchmark tests. 

ጠጠ ሮህ ክድ the user's evaluation criteria, the vendor has 
a much better understanding of what type of system must be 
ENODOSed, and proposes accordingly. if the user identifies 
E © having greater importance than run time, the vendor 
may tailor the proposal to a smaller system with fewer time- 
Saving devices. If the user indicates, by value assignment, 
that reliability is important, the vendor can propose a 
system complete with error detection and correction features, 
Moreover, he can develop alternative proposals and determine 
with some degree of accuracy which alternative most nearly 
fits the user's requirements. 

3. The vendor has a chance to comment on criteria, prefer- 
mee carly in the game, and identify areas in which conditions 
may not be fair or meaningful. It is always possible that the 
user may be willing to modify these conditions, 

When the user discloses so completely the evaluation 
technique he plans to use, he benefits in the following four 


ways: 
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RC he proper vendors are biddins. Those whose 
system መ nov possibly win will see the handwriting on 
the wall and drop out. Overall, fewer systems may be 
proposed, but those proposed should all more nearly iit tne 
user's requirements. 

meee The need to evaluate a multitude of proposals should 
be minimal because the vendor would have been encouraged To 
Sëtze inappropriate alternative proposals. The user is 
Baus in a position to receive system proposals as nearly 
suited to his wishes as vendors can make them, 

3. The user can now receive free expert advice on the 
stated system requirements and on the evaluation method to 
be used. Most vendor comments on the evaluation method will 
minimize the importance of features and abilities their equip- 
ment does not have and emphasize tne value of those their 
equipment has. But also there will be valuable suggestions 
on better evaluation methods or in other meaningful areas 
that were not intended to be evaluated originally.  Sugges- 
tions of this kind may help the users to get a system better 
suited to their desires and needs. 

SYSTEM SUPPORT. The kind and extent of vendor support 
necessary for attainment of all system objectives should be 
stated, and may extend beyond maintenance and training needs. 
Programming assistance, special subroutines, and other special 
requirements for which the user has a genuine need, may be 


Hevea cdetinedwas prerequisite conditions. 
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BENCHMARK DATA, The benchmark programs to be used as 
Enem description and validation should ve supplied, along 
with sample data and answers by the prospective user. 

EE CONFERENCE DATES. Ihe bidders' conference is 
a formal presentation to the vendor, by the user, of his 
system requirements. This conference should te neld a weex 
or two after the vendors nave received the specifications 
package and have nad a chance to review the system require- 
ments and desires. At. the conference, any questions on the 
techniques to be used in evaluating the proposal snould be 
discussed and resolved. The date for the conference and a 
general explanation of its purpose should be included in the 
request for proposal (RFP), 

CHECK-IN DATES. Check-in dates are dates, determined by 
٣۰۰٠-۰٠٠۹ ٣ which each vendor should indicate whether or not 
he is still engaged in preparing a proposal; if so, if the 
vendor still has any guestions, he should ask tnem at this 
Dame. | | 

PROVISIONS FOR HANDLING QUESTIONS. Since questions will 
arise throughout the proposal period, the user should state 
Provisions for handling them, not only at the beginning, but 
at the various stages of the selection, 

PROPOSAL DUE DATES. The proposal due dates should be 
explained, along with a clear description of what will happen 
to late proposals. 

VENDORS' DEMONSTRATIONS AND PRESENTATIONS. Some time 


after the submission of the proposals, the vendors should be 
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afforded the opportunity to tell why they submitted as they 
did. Having acquired benchmarks, the vendors should be 
required to provide demonstrations. 

CONTRACTING CONDITIONS. The vendor should be informed 
that any promises he makes will be written into the contract 
and that the contract will have to be signed by a corporate 
5۶۰.۲۱161. 

AWARDING AND DEBRIEFING DATES. Dates should be set for 
the awarding of the contract and debriefing vendors. 

GENERAL COMMENTS. This section should contain any 
۱۳۱۰ ۱ ۱۸۵ 10۳5 on the format oí the proposal, such as arrange- 
Heme Oot information with the proposal, number of copies to 
۶٣۰۱۔۱666,‎ ۹8,343 so on. The purpose of this section is to 
make selection easier for the user by keeping things as 
uniform as possible among the several proposals. 

The specifications package is the first official state- 
ment of the details that the vendor will have concerning the 
Beer's problems. ‘Thus the specifications must be stated 
clearly, questions asked by the user should be meaningful, 
and the evaluation method to be used well defined. The 
peeci fications package establishes the vendors' first thoughts 
On the system. These thoughts are also important to the user 
because if the vendor is given to believe or suspect that the 
equipment request is of low quality, or that the system (or 
evaluation method) is poor, he may abstain from bidding. The 
vendor who decides not to bid may be withholding the system 


Best SUited for the task. 
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۲ VENDOR CONTACT 

Ren ita the locikc-out policy, if will be necessary for 
the company personnel to come into contact with tne vendors 
in several ways. First, tnere should be some time set aside 
when the vendors and the company personnel can sit down and 
Nu cuss the specifications package. "nis will occur normally 
during a bidder conference. There also is a continuing need 
to answer the vendors' questions as they arise. The vendors 
should be allowed to make some form of presentation after 
Eu ሮድ their proposals, and they should be required to 
provide any demonstrations called for in the specifications 
package. Finally, there will be the pleasure of telling some 
vendor that he has won and the necessity of telling the other 
vendors why they lost. 

x. Bidders! Conference and Questions 

A bidders' conference is not always necessary espe- 

cially if the specifications are simple and straightforward. 
However, if a bidders! conference is to be held, it is 
essential that the user be well prepared for it. The best 
preparation 1s a good set of specifications with meaningful 
benchmarks. The evaluation procedure should accurately 
reflect the user's requirements in system capability. At the 
bidders' conference, the user should have his best people 
available to explain and define the requirements and evaluation 
procedures, The user must consider all questions with an oren 


mind. Where there is a possibility that some requirement or 
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procedure might be wrong, the question should be held open 
until a proper answer can be found. Answers "off the top of 
the head" are not sufficient. 

Bo eausscions raised qurine che conference should be 
Eme reo as quickly as possible. All the questions asked 
should be studied to determine whether they affect only the 
Ens vendor or all vendors. If the latter, then the point 
Mould be clarified for all vendors. 

The bidders' conferenee will not dispel all the 
vendors' questions; even if it did, others would soon appear. 
The best means of enabling vendors to ask questions is a 
telephone call to the asking vendor, tnen written documenta- 
Prom Mailed to ail vendors. 

2. Vendors! Demonstrations and Presentations 

When the specifications package proposals require 
that the vendors demonstrate their computer systems, the dates 
for these demonstrations should be left up to the vendors es 
much as possible. In running a benchmark program or other 
demonstration for validation purposes, the vendor may need to 
obtain the proposed components from several other systems 
Peemeereadied for shipment, After deriving the timing informa- 
tion required from these demonstrations, the vendor then must 
decide whether to release the UNES for shipment as 
planned, and perhaps not be able to reassemble the proposed 
stem in time for an official demonstration, or to hold the 
components, thus tying up several systems which otherwise 


could be shipped. 
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W JE permi ceed co demonstrate whenever 
۲ ready, he can be spared such a difficult decision. 
The user also benefits by an early demonstration, which 
affords nim an earlier check on the vendor. If things do 
not go according to the vendor's plans and the demonstration 
does not demonstrate quite what the vendor said it would, 
Ene vendor has an opportunity to take these findings into 
consideration when preparing his proposal. One or two weeks 
after the vendors have submitted their proposals, they snould 
be given an opportunity to make a formal presentation. 

۲ Contracting and Debriefing 

Normally many statements made in the proposal cannot 
Re verified. Any which had a bearing on the winning proposal's 
Ene won should be written into the contract covering that 
system. Statements of this sort are those dealing with either 
tne mandatory conditions or the desirable features requested. 
It should be made clear that the contract will require the 
Signature of a corporate official. Without such a signature, 
۱۳ د‎ 2200 1S no Stronger than tne position cf the Signer. 

Salesmen, under a strong emotion, have been known to 
State anything! Corporate officials, when they are signing 
K. Manes, are pledging their corporation's funds. This 
s ce reinforced by penalty clauses contained in the contract 
which cover late delivery, failure to deliver, and the like. 
These penalty clauses, as well as the general. writing of the 


Eomnvrpacr. Should be handled by the company's legal staff. 
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Immediately after a formal announcement of vendor selection, 
a debriefing should be arranged to ensure that none of the 
losing vendors is left for long in the position of knowing 
they lost, but not knowing why. 

The debriefing is something that should be handled 
۱۲۲ ۱ 1 ۷3 2۵۰ Each vendor should be told exactly why he did 
not receive the contract. If the selection was handled 
openly, the vendor should have little dispute since he was 
aware of how his proposal stacked up. Usually, the major 
point of dispute will be centered on adjustments that were 
made to his proposal. If the vendor was made aware of, and 
agreed to, the adjustments during the validation phase of the 
selection, there should be little he can say. Table 25 gives 
West Of the factors which must be included in any 
contract [Tarussell 1576۳٣٠٠٠٠٦٠٠٠١٦ 66٦ 1972, Chorafas 


1967, Webster and Johnson 1976, Yearsley and Graham 1973]. 
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ADP CONTRACTURAL FACTORS 


Equipment inventory (model number, etc.) 
Mes (rental terms or lease, etc.) 
Performance 

Delivery 

Installation 

Snsport 

Maintenance 

warranty 

Attachment (of other manufacturer's equipment) 
Permeability 

Training 

iSebñical specification 


Beeeprance 
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IX. COMPUTER PERFORMANCE MEASUREMENT AND EVALUATION 


Data processing can be viewed as a production facility 
which is to satisfy the reeds of its users, Tyrical users 
may be the payroll department staff producing paychecks, 
programmers debugging programs, and engineers solving techni- 
cal problems. The needs of these users are to have their 
jobs processed correctly, on time, and economically. System 
prmance evaluation is an attempt to determine how well a 
EI system is meeting or may be expected to meet specific 
processing requirements at specific interfaces. This diffi- 
cult task is more easily carried out as three distinct 
evaluation activities: 

Ie Cost Activity. The objective of this activity is 
bosdswmermine tne one time and recurring costs from the first 
planning stage through the replacement of tne system, 

Emmeline Judgement Activity. The objective of this activity 
is to evaluate the nonquantifiable factors such as: 

What improvements can the vendor be expected to make in his 
produce line during the next five years? 

How will these benefit the company? 

What level of system component maintenance can be expected? 

ER TRE Cerfommance Evaluation Activity. The objectives of 
tnis activity are to develop meaningful, quantitative measures 
Of how the system may be expected to complete a day's work in 


a day and estimate the unused capability available. 


30 





The main interes; of this chapter is only the performance 


aon activity, and it will be discussed in more detail 
in the rest of this chapter. Before system evaluation can 
il three inputs are required: 

۳ => description of a specific system to be evaluated. 

Bine specific processing requirements, 

۳ ۱۳5 identification of each interface across which the 
ERR Mm is to be evaluated. 

Mie first input demands that all the components of the 
system be specified prior to the start of the evaluation 
because the evaluation process is being applied to the entire 
system [Stimler 1974, Rosen 1976]. Every hardware device and 
precisely how that device is to be connected in the system 
must be specified. The operating system, user programs, and 
job scheduling procedures also need to be specified. This 
includes identifying when jobs will be run and which Jobs 
are to be multiprogrammed. The accuracy of an evaluation 
depends directly upon how well the system is defined. When 
different configurations are to be evaluated, each must be 
evaluated separately. 

Ihe specific processing requirements must be identified 
for the second input. Typical of the information needed here 
is the work load to be processed and turn around times to be 
met by month, week, day and hour. Periods of heaviest proc- 
essing loads and shortest turn around time requirements are 
of special importance. One processing requirement to eval- 


uate would be the maximum number of transactions per hour the 
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terminal operator can enter and have the computer-generated 
outputs arrive within the required response and turn around 
0 ٠١ 

The evaluation interface between each user of the system 
and the rest of the system must be identified for tne third 
essential input. It is usually a human-system interface, 
such as between a terminal user and the terminal device in 
evaluating a real time system. 

The effectiveness of a system can also be described in 
terms of the capability to process a given workload, and the 
capability to meet time requirements. of individual users. 

The efficiency is measured by internal delays and utiliza- 
tion of individual system components versus demand.  Zífec- 
tiveness measures are the prime performance measures. Values 
of these measures can be assessed from observations made act 
the external side of the evaluation interface: they are what 
is seen by the system users. These measures are frequently 
called external performance measures [Svobodova 1976]. Effi- 
5 ርኛ íis an Eerma] factor; values of efficiency measures 
usually must be obtained from within the system. These 

5 ۰۰76ح‎ ad in identifying problems that diminish system 
effectiveness. 

Examples of both external and internal performance 
measures are given in Table 26. Performance measures are 
most frequently expressed as mean values. In many cases, 
Sab values are clearly inadeguate measures of system perform- 


ance. For example, if the variance of the response time of 
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TABLE 26-a 


EXAMPLES OF PERFORMANCE MEASURES 


Performance Measure le 80 


SYSTEM EFFECTIVENESS 


Throughput Amount of useful work completed per 
ከ ከው ን ከ Sw: slven workload 


Relative Elapsed time required to vrocess 

Throughput given workload on system Sl/elapsed 
time required to process the same 
workload on system 32 


fapebilit; Maximum amount of useful work that 
(Capacity can be performed per unit of time 
with given workload 


Turnaround Time Elapsed time between submitting a 
job to a system and receiving the 
OUTPUT 

Response Time Turnaround time of requests and 


transactions in an interactive or 
a real time system 


Availability Percentage of time a system is 
available to users 


———MM M M M ‪-ث ں -. 0 ں21 َٹب‎ 0 7000 
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TABLE 26-b 


EXAMPLES OF PERFORMANCE MEASURES 


Performance Measure 


Description 


SYSTEM EFFICIENCY 


External Delay Factor 


Elapsed Time 
Multiprogramming 
Factor (ETMF) 


pun Factor 


EB Productivity 


Component Overiap 


SYS tem Utility 
Overhead 


Internal Delay Factor 


Reaction Time 


Hait Lime For I/O 
Wa To Time For CPU 


Page Fault Frequency 


Job turnaround time/job processing 
time 


Turnaround time of a job under 
multiprogramming/turnaround time of 
this Job when it is the only job in 
the system 


Total system time needed to execute 

a set of jobs under multiprogramming/ 
total system time needed to execute 
the same set sequentially 


Percentage of time a CPU is doing 
useful work (used as a measure of 
throughput) 


Percentage of time two or more 
System components operate 
Simultaneously 


Weighted sum of utilization of 
system resources 


Percentage of CPU time required by 
the operating system 


Processing time of a job under 
multiprogramming/processing time of 
this job when it is the only job in 
the system 


Time between entering the last 
character on a terminal or receiving 
the input in the system and 
26٣3972٣1۳ ۶۰٠۰۱۱٦ 6133 650 


Elapsed time required to process an 
I/O task 


Elapsed time required to process a 
CPU task 


Number of page faults per unit of time 
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an interactive system is large, the user is likely to be 
dissatisfied with the system performance even if the mean 
response time is reasonably short. Thus, if the exact dis- 
tribution of the response time is not known, at least the 
variance of the response time ought to be considered as a 
E mance measure in addition to the mean response time. 
A good measure of performance of an interactive system is 
the percentile response time. N percentile response time is 
defined as the time limit that guarantees that the response 
times of N percentage of all requests are shorter than this 
limit [Sekino 1972]. One cannot expect the response time 
for very involved requests to be as short as the response 
time for trivial requests. Thus, response time (percentile 
response time) should be assessed separately for different 
classes of requests. 

Performance measures can be specified only with respect 
ከታ ወከ ype and the purpose of the evaluated system, its 
workload, and the purpose of evaluation [Svobodova 1976]. 
Performance measures must be well defined since they set a 
framework for the entire evaluation process. 

Having selected performance measures, the crucial problem 
ls to determine how these performance measures depend on the 
system workload and the system structure. An understanding 
of such a relationship is essential if performance optimiza- 
IS ጵ.፡  ፡ የ. to be constructive, but it IS also important 


when selecting a new computer system. An expression of this 





relationship is the performance model of the system, The 
performance model is the ultimate goal of system analysis 
[Svobodova 1976; Rosen 1976]. 
The values of performance measures are determined by a 
combination of the following: 
1. Measurement 
2. Analysis 
۳ Simulation 
The most accurate values are obtained when the system is 
measured under its real workload. Because of the variability 
or unavailability of the real workload, it is often necessary 
to design an arvificial reproducible workload and measure the 
system performance against this artificial workload. when- 
ever evaluating a system that has not yet been implemented 
or is otherwise unavailable for measurement, it is necessary 
to develop a functional model of that system. The values of 
performance measures are then obtained either by analytical 
means or by simulation. 
Measurement and modeling are complementary processes in 
that: 
1. a model provides a framework for measurement, 
2. measurement provides data for validating the model, 
3۰٠۲۳۰۱۹4917 aids: in. testing hypothesis and finding 
solutions to performance problems, and 
4, correctness of model predictions is finally verified by 


measurement. 
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A. THE SYSTEM PERFORMANCE EVALUATION METHODOLOGY 
The system performance evaluation methodology requires 
the successful carrying out of the following Six steps: 
1. define the technical terms used, 
۲ establish performance criteria, 
3. acquire the specific input data needed for eacn evalu- 
a4 BION, 
4. analyze the performance of the system being evaluated, 
5. use appropriate evaluation aids, and 
6. document the evaluation results. 
Bach step will be discussed in detail. 
Define the Technical Terms Used 
To be able to meaningfully answer the question "What 
is the performance of the system?" with "It is operating at 
۳٠۰۰٠٠ of its capability during the third shift", it is 
essential that voth those asking and those answering the 
questions have a common understanding of all technical terms 
used, such as performance, capability, and system. Since 
there is no industry-wide accepted dictionary of data proc- 
essing terms, different practitioners use the same word to 
have different technical meanings and different words to have 
the same meaning. If meaningful numerical expressions for 
performance are desired, this problem nas to be overcome. 
The following guideline could be very satisfactory. 
awin the assumption that no English word or 
group of words has any inherent technical meaning.  Perform- 


ance, system, or time sharing assume technical meaning only 
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after those intending to communicate define each word or 
combination of words they intend to use. Further, the 
essential criteria for the definitions are: 

a. They are clearly understood by those using them. 

Pare definitions should be operational, i.e., permit 
physical measurement to arrive at numerical values. 

e. New definitions should not be originated for commonly 
accepted terms which meet the first two criteria. 

۱۳۳ ۰ ۱۵ ۲ eriterron ofzelaprxcyrjs extremely difficult 
to achieve, The enormity of the task can begin to be appre- 
ciated when it is considered that the single word system is 
being used as a symbol to communicate the idea of a complex, 
operational organization of men, machines, orograms, and 
procedures. Therefore, it is suggested that a definition 
which is adequate for effective communication should be 
developed and used at the time it is needed rather than wait 
for a perfect or a standard definition which might be avail- 
able after the immediate need has passed [Stimler 1972]. 

The operational criteria require that such definitions 
as througnput, response time, and capability permit physical 
measurement to arrive at numerical values when the system is 
operational. The third criterion is intended to reduce the 
ille OL different definitions for the same concept. 

2, Establish Performance Criteria 

A performance criterion is a performance standard 

with which comparisons can be made. For example, the cri- 


terion for the throughput of a system is here defined as the 
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tada ta processing work successfully completed during an 
ton period, "Throuchput, like every other criterion 
to be defined, is applicable to all classes of data processing 
systems. However, since the unit of data processing work is 
Seeererent for each class, the unit of work unique to the 
class of system being evaluated must be used. 
3. Acquire The Specific Input Data Needed For Each 
Evaluation 
Input data needed include the exact way each compo- 
Meme 2S cOnnected, the configuration and characteristics of 
all hardware and software components, and so on. 
!. Analyze The Performance Of The System Being Evaluated 
A "pencil and paper" analysis is the first essential 
level of analysis. This procedure is: 
a. understand, in deptn, the operation of the systen, 
b. understand, in depth, the operation of the system 
components, 
c. set up a model of the system, 
d. determine and keep in the model only the significant 
components, 
T erlie mathematical relationships for each of the 
criteria to be used in the evaluation, 
f. insert system characteristics into the mathematical 
relationships and derive the reguired results, 
E. | perform sensitivity tests which indicate the relative 
effect each component has on performance, and 


h. prepare conclusions and recommendations. 
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5. Use Appropriate Evaluation Aids 
Simulations, benchmarking, and resource utilization 
monitors are available aids for the evaluation process. 
These aids, properly used, can provide cost-effective supple- 
ments to the pencil and paper analysis. Improperly used, 
these aids can be expensive, time consuming, and misleading. 
6. Document The Evaluation Results 
One of the essential results of any performance 
evaluation and performance improvement effort is to document 
what was done, conclusions reached, and recommendations made, 
Documentation is an essential step in the methodology [Rosen 


1976, Stimler 1974). 


B. THE CONTROL OF SYSTEM PERFORMANCE 
Listing all parameters that affect computer system 
performance would be an exceedingly difficult task. The 
performance of a computer system with respect to a specific 
٠۰.۰۰.۹106 is a function of: 
m y Stem configuration. 
2. Resource management policies of the operating system, 
3. Efficiency of system programs. 
4 Effectiveness of the instruction set processor. 
2. Speed of hardware components. 
Performance characteristics are shaped in three stages: 
l. system design, 
2. System implementation, and 


3. matching the system to a given workload. 
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Most of the performance evaluation and optimization 
efforts are presented in stage three, because each informa- 
tion processing system handles a different workload. This 
stage is concerned mainly with system configuration and 
resource management; that is, allocation and scheduling of 
Processor Memory Switch (PMS) components. 

Performance of a particular computer system installation 
can be controlled in several different ways: 

l. Adjustment of system control parameters, 

EE One or modification of resource management policies, 

3. Balancing the distribution or load among system com- 
ponents through system reconfiguration (changes in the 
assignment of peripheral devices to channels or the assign- 
ment of files to physical storage devices, changes in the 
distribution of software components in the system memory 
hierarchy, etc.), and 

4. Replacement or modification of system components. 

As long as the user interface does not change, the system 
does not change to the user; only the performance does, 
However, configuration changes and software changes result 
ina new system, a system that has to be designed, analyzed, 
implemented, tested, and documented. Control parameters can 
be changed as needed without having to test the system 
lona Il. Table 27 lists some system parameters that 
can be used to control system performance. Control para- 
meters can be set either before tne system is started, or 


modified if necessary during system operation [Svobodova 1976]. 
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EXAMPLES 


Control Parameter 


TA BERTZ 


OF CONTROL PARAMETERS 


Description 





Quantum Size 


0000521121 Priority 


Degree of 
Multiprogramming 


Memory Partition Size 


Window Size 


Maximum Allowed 
Paging Rate 


Page Survival Index 


Number of Simultaneous 
Users 


Device-to-Channel 
Assignment 


Time quantum in which the CPU of a 
time-sharing system is allocated to 
jobs 


Priority based on the demands of a 
job and services already received 


Number of jobs that are simultaneously 
in the main memory and thus eligible 
to use the CPU 


Amount of main memory allocated to 
a single job ۱ 


Time interval for determining the 
working set of a job 


Maximum allowed paging rate in a 
demand paging system | 


Number of CPU bursts received by a 
program before an unreferenced page 
is removed from main memory 


Number of terminal users logged onto 
the system 


Assignment of I/O devices to avail- 
able channels 





176 





0 d 





In the latter case, changes may have to be induced by the 
operator, or control parameters can be changed automatically 
mm response to changing user requirements. 

Turnaround time or response time measures not only the 
system performance but also the quality of the program that 
constitutes the job. Performance improvement with respect 
ከጠ ከ Specific application ought to be approached from both 
sides: reducing the amount of work required by the applica- 
tion, and improving the efficiency of the system [ferrari 
1975; Hatfield 1971]. 

Sometimes improvement of system performance with respect 
to a particular performance measure is possible only at the 
cost of reducing performance with respect to some other 
measures. The qualitative value of a specific level of a 
performance measure is the user's preference for this level. 
Performance trade-offs can be resolved only if the relative 
preferences for different levels of different performance 
measures are known. Determination of the preferred 
conbination is the basic problem of decision theory. 

The system performance can also be assessed in terms of 
the cost of using the system. The cost of using the system 
ግይ unccion Of the system cost and the cost of the program- 
mer. The higher the throughput, the lower is the system cost 
per unit of work. The shorter the response time, the less 
the programmer's time is wasted waiting for response and the 
lower is the cost of programming. As the system approaches 


its capacity (maximum throughput), the response time suffers. 
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A proper balance between throughput and response time has to 
p-mestablished such that the cost of using the system is 
amimized, 

Em important factor that influences the productivity of 
a system user is the ease of using the system for a specific 
۰۰۰٠٠۰٠۰٠۰ ٠۰ ዎር በር This factor has received more attention under 
the label "human engineering". The response time belongs to 
the category of human-oriented considerations; however, it 
is neither the only important consideration nor the most 
important consideration. Ease of use and performance are 
frequently conflicting design requirements. Since both of 
لد‎ 3 12060015 can make a user's task either satisfying or 
frustrating, there is no simple rule as to how to resolve 
۱9 1۱0۲: 1 6 1۵ | Svobodova 1976]. 

In general, several different system models are used 
during various stages of a performance evaluation project. 
These models can be divided into three general classes: 

۱۰۰۷۷۶۱۰١ 78نا‎ models 
2. Functional models 
۳۱۳۵ 1 9۱۱۵ 1 models, used in performance analysis, can 
ددن‎ ያሮ into four groups: 
a. Flowchart models 
b. Finite-state models 
C. Parallel nets 
d. Queuing models 


3, Performance models 


I 





Deccsucetural model deseribes individual system components 
Mea neir connections. Such a model provides a useful inter- 
face between the real system and a more abstract model. 
Structural models are most frequently represented by block 
diagrams. The level of detail in a block diagram can easily 
Epica since individual blocks can in turn be further 
laid down as self-contained block diagrams. Block diagrams 
generally show the paths of data flow as well as control 
Flow, but they do not specify the conditions governing this 
flow. Thus, block diagrams are suitable only as the first 
general level description of the system under study. 

A functional model describes how a system operates. It 
defines the system such that the system can be analyzed 
mathematically or studied empirically. 

ENcwehart models are suitable for studying Program 
efficiency and execution time requirements. A flowchart 
model is a directed graph model where the nodes represent 
computational tasks and the arcs show the possible flow of 
control between tasks. Flowchart models of system components 
and users' programs can be used as building elements of a 
System model, tied together by a mechanism that simulates 
System resource allocation and scheduling [Anderson 1976]. 

A finite-state model can be used for analysis of utiliza- 
tion of computer system resources. A finite-state model can 
be represented by a directed grapn and, in this case, the 


nodes represent the state of the system; the arcs represent 
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the transitions between states. The system state is composed 
of the states of individual system components and it thus 
relects concurrency of system operations [Coop 1971]. 

Parallel nets are directed graphs made of two difíerent 
Eus of nodes: transitions and places. Places with ares 
directed into a transition are the conditions that must be 
ከግ ር ከመ concurrently if this transition is to occur. Such 
nets were found to be a useful aid in the design and imple- 
mentation of a simulation model and in a planning of measure- 
ment experiments. 

۱۳۱ azaueuing models concept, a computer system is a set 
of resources and queues for these resources, When a job 
enters the system, it is placed in one of the queues where it 
waits until the requested resource becomes available. After 
a request has been processed, a job either leaves the system 
or enters some queue again. Queuing models emphasize the 
flow of jobs through the system, but they also enable one to 
observe the state of the system, These models are the most 
widely used models in computer performance analysis. 

A performance model formulates the dependence of perform- 
ance on the system workload and the system structure. It is 
derived by analysis of a functional model for a specific 


model of workload. 


٠۰۰۰٠٠٠٠ ٣٢٠٦ ٣۲٢۲ ۸ ۲۲01١17۰0۲۷ DATA PROGESSENGZSYSTEMS 
Commercial data processing systems can be generally 
classified into three classes: 


l. Batch systems 
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2. Real time systems 
۳۳ 1۵ ۵ دی‎ 6 time sharing systems 
ell 01355 fits the definition of a data processing 

system in that it is an organization of hardware, software, 
user programs, procedures, and people capable of transformirg 
Enmecedfjed inputs into specified outputs. However, from both 
the performance evaluation and the system design viewpoints, 
each is sufficiently different to require a separate classi- 
fication. An essential difference among the classes is the 
unit of data processing work. Table 28 shows the cnaracter- 
istics used to determine the classification of a system 


[Stimler 1974; Rosen 1976; Wooldridge 1973]. 


D. THE UNIT OF DATA PROCESSING WORK 

The unit of work for each class of system is briefly 
described in this section. 

1. Batch Systems 

ime processing of a job, as identified in the job 

logging routine, can be used as the unit of messure for batch 
processing work. Jobs vary widely in the amount of input, 
processing required, storage used, and output generated. 
The رہ‎ e jobs may also vary during different 
parts of each day. The specific jobs and input data frequently 
vary with day of the week, week of the month, and month of 
the year. For evaluation it is necessary to determine a 
representative workload profile. In many batch processing 
facilities a full month is needed to process an acceptable 
representative work load. In engineering facilities a week 


may be enough. 
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2. Real Time System 

The processing of a transaction is the unit of work 
ኔን ፡ cnis class of system. Ine processing of a transaction 
ከጠብ ይጋ ቢር receipt of the input, its processing, and trans- 
mission of all required outputs. Each transaction is com- 
pleted in seconds and there usually are a limited number of 
different transactions a user can input. There may be from 
two to fifteen different transaction types. The combination 
of the limited number of different transactions and the 
snort processing time per transaction permits meaningful 
evaluation of real time systems in periods as small as ten 
to fifteen minutes. Inquiry and message are commonly used 
۲۰ denote a real time input and output. Transaction is used 
to include these terms. 

3. Interactive Time-Sharing Systems 

An interactive time-sharing system provides each 

terminal user with essentially all the system capabilities 
he would have at the computer console except that he must 
share the computer resources with other users, This capa- 
bility means that one terminal user can be comriling and 
debugging a new program, another running a program for the 
first time, another building a new data base, and another 
generating complex inquiries of a data base. Some of these 
inputs require responses in seconds, others in minutes. 
Normal batch production jobs frequently are run in the back- 


ground when resources are available. 
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The unit of work is the sum of transactions processed 
pie batch jobs processed. To make meaningful evaluations 
BmemeOmoarisSons it is necessary to use the identical mix of 
۲۱ ۱۵ each calculation [Stimler ٠۷۰۶۰۰٠۰ ٠٠۰٦6۹4618 1976, 
205 1973] . Basic definitions of performance criteria 
are presented in Appendix À in the form of simple equations 
to facilitate the calculation of numerical values. 

Appendices B and C of this document provide sample 
checklists for hardware and software respectively. They can 
be applied to computer systems or their major parts (CPU, 
peripheral units) to obtain some criteria for performance 
evaluation and comparison. The data obtained can also be 
used in the apolications of the computer selection and 


evaluation methodologies. 
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X. CONCLUSION AND RECOMMENDATIONS 


Today's great emphasis on computer systems is an impor- 
tant reason for management's increased concern about major 
expenditures. Talk of acquiring a new computer system causes 
considerable interest at many levels of management.  Manage- 
ment has e major role in each of the three principal phases 
of acquisition: systems analysis and design, selection, and 
۱۳۱ 02 113 

Management must appoint good people ard support them for 
the acquisition effort for the new system. The individuals 
appointed to the acquisition team must be able to communicate 
with management to ascertain management's needs and desires. 
Management must also guide and direct the team. They should 
be informed as to tbe approaches that might be taken in 
acquiring the computer system. The establishment of realistic 
milestones must be required by the managemenct. 

The responsibilities of the management in the three 
phases of acquisition, mentioned above, can be stated as the 
followings. 

A. In the systems analysis and design phase of acquisition, 
management: 
l. should discourage pioneering with the new system, 
2 202۱۰۰۰۷۰۰۰۰۶۰ ت0‎ 176 systems life forecasting, 
3. should demand system design alternatives, 
H 


should require meaningful economic justification, 
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IS NOUS ist on common language programming, 

6. must assure the availability of procurement funds, 

y must review the system requirements, 

8. should require that system specifications (not 
equipment specifications) be issued to the vendors. 

B. In the selection phase of acquisition, management: 

۲۳ must reguire competitive specifications. 

2. should encourage the issuance of a presolicitation 
letter to assure that the competitive specifications sought 
in the previous step have been achieved. An advanced copy 
of these specifications should be sent to prospective vendors 
for their review. The vehicle for sending the specifications 
ps e vendors is a presolieitation letter. 

3. should require the.establishment of a formal 
Selection Plan. 

4. should insist that, for any medium to large scale 
procurement, representative benchmark mixes be used for 
workload representation and validation. 

ER should require a formalized evaluation process for 
system selection. 

6. should require the use of System Life Costing in 
the evaluation process. 

7. Should review the complete Solicitation Document 
and Selection Plan before the Solicitation Document is 
released to vendors. 


8. must live by the Selection Plan. 
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e installation phase ot acquisition, management: 

Sonia require tmab ali activities leading to 
٠ ء5٦18‎ 61605 be scheduled. 

2. must assure that a formal system acceptance test is 
۶ Led. 

٠۶۰ت‎ ۱٠۷٠۷٠۰٣٠٣ thay Gherouem, complete documentation be 
Brevided., 

The cost of the proposed system cannot be ignored, there- 
fore when choosing a selection methodology basic elements must 
5 ۷6۰ These are 

1. the assessment of the value of vendors' offerings to the 
buyer (EVALUATION), and the 
2. validation of the vendors' claims (VALIDATION). 
lmplementing sophisticated evaluation and validation tech- 
niques used for selecting large or medium-sized computers can 
easily tie up three or more people for one year or more. The 
cost of their time, travel, computer use, and other expenses 
can easily exceed $50,000. That expenditure may be ۵ ۵ ۲ dre 
for a $5,000,000 computer system, or even for a $500,000 one, 
but it makes little sense for the buyer to spend 350,000 to 
decide how best to spend another $50,000 for a small computer 
system. 
The buyer must face the fact that it is necessary to invest 
a certain minimal amount just to play the computer selection 
game. He must know how he plans to use the desired computer 
system. Determining the need for a minicomputer may take one 
person six months and cost $10,000. The same task with respect 


to the need for a large computer may take a team of five people 


one year and cost $100,000. 
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۱۳ adzaneinge technology has drastically reduced 
the cost of computer hardware, and at the same time inflation, 
coupled with the demand for more complex computer applications, 
has increased the cost of the human effort associated with 
computer services, namely, the system study, the selection 
peeeess, programming, and operation. 

If the buyer goes the competitive route because of regu- 
lations or otherwise, he should become concerned with ways to 
minimize the cost of the selection process. Since the cost 
of preparing (writing) the Request for Proposals or Solici- 
tation Document is usually somewhat fixed and minimal (once 
a good document has been found for use as a model), only the 
evaluation and validation processes provide opportunities for 
ی۰٣٦١‎ costs significantly. 

Simplification should not be interpreted to mean that it 
is necessary to use an inferior selection methodology which: 

1. May fail to consider all the desired (but not mandatory) 
items or features, 

2. May not facilitate establishing meaningful and under- 
standable relative values between the desirable items, 

3. Does not permit disclosing the relative value of the 
desired items in the request for proposal (system), 

4. Fails to incorporate systems life costing. 

These failings can be avoided by using the simplified 
version of the Cost-Value or Requirements Costing evaluation 
methodology. The more time spent on establishing the values 


of the desirable features, the better the technique becomes. 
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In view of the fact that benchmarking is the only vallda- 
buen process which is really defendable, the cost of bench- 
marking must be cut, The high cost of benchmarking has 
fas been at the top of any vendor's list of complaints 
about competitive bidding. The biggest cost factors to the 
vendors are the cost of debugging the benchmark programs and 
the cost of pulling together and holding a system of the 
type necessary to demonstrate the running of the benchmark 
programs. Benchmarking small systems rarely involves complex 
emS, so the only problem is that the vendors will not 
bid on small systems if too much is expected of them in 
debugging or running the benchmark programs. 

monmiurtnier Simplify whe vatieation process, demonstrating 
the benchmark mix can be required of only the winning vendor. 
This not only reduces the vendors! costs, but also greatly 
reduces the buyer's costs, for demonstrations quickly consume 
man-days and travel dollars. 

The particulars (e.g., government/private sector, expected 
system cost, applications, etc.) of the buyer's situation 
must dictate the means to be used by him for simplifying the 
maeceaure for selecting a small computer which satisfies 


his needs. 
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APPENDIX A 
PERFORMANCE CRITERIA 


EQ. 1 


Total data processing work successfully 
Throughput = completed during an evaluation period 


EQ. 2 


Throughput 
Wall clock system time 
(in hrs., min. or sec. 
to process the throughput 


Throughput rate - 


EQ. 3 


Throughput of a repre- 
Average throughput -= sentative workload 
"c Wall clock system time 
(in hrs., min. or sec.) 
to process the repre- 
sentative workload 


EQ. 3 for real time systems 


Transactions successfully 
completed in a represen- 
Average throughput = دق‎ we 89 
Ne ei z Wall clock system time 
(usually in min. or sec.) 
to process that workload 


EQ. 3 for batch systems 


Jobs successfully completed in 
Average throughput . 3 representative workload 
rate ali edocr system time (in hrs. 


to process that workload 
EQ. 3 for interactive time-sharing systems 


(Representative short job workload 
+ representative long job workload) 
successfully completed during an 
RO nrousheut - evaluacionp od | 
ርን 5 ር اوہ ا‎ wall clock system hrs. expended 
bo process that ۱١۵2۱99803 
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EQ. 3 for real time systens multiprogramming 
background batch jobs 


(Representative transaction work- 
load + representative batch work- 
load) successfully completed 
during an evaluation period 

Wall clock system hrs. expended 

to process tnat workload 


average türougürput = 
rate 


EQ. 4 


Maximum achievable average throughput rate 
Capability - regardless of the timeliness of outputs 
cover a given time period 


Hus. 5 


Maximum achievable average 
Operational capability = throughput rate while meeting 
timeliness requirements 


EQ. 6 


Average throughput rate 


ve t — for evaluation l ^ | 
eet Ve «hroughput Average throughput rate 


Dore salta tion 2 


uo 7 


Capability for evaluation 1 


Relative capability Capability for evaluation 2 


EQ. 8 
average 
(Capability - throughput) 100 
Percent unusad = Ha 
capability Capability 


EQ. 9 


Relative 


RuccneEthroughput . _ 
rate change š a i O 


BO. 10 

Elapsed time in hrs., min. or sec. between 
Tu 1 ERR Om ns charaererior first. input at 
cS en = input interface and arrival of last character 


En du: cupons required, by that Input at 
output interface 
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m 


Elapsed time in sec. between arrival of 
last character of an input transaction 

Response time = at input Interface and arrival of first 
Character ۲٦1٦٦6۳۳ 0116 0106 8 6.011 
interface 


Bee 12 
Turnaround time of a specific job 


processed in a specific multiprogrammed 
Elapsed time environment 


TS lication = “Turnaround time of the same job 

factor (ETMF) processed as the only job in the 
Same system 

EQ. 13 

Equivalent throughput rate of N independent systems 


= Sum of throughput rates of N systems 


= Throughput rate + Throughput rate +... + Throughput rate 
of system 1 of system 2 | of system N 


Equivalent capability of N independent systems 


Sum of capabilities of N systems 


Capability of + Capability of + ... + Capability of 
system 1 system 2 system N 
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APPENDIX B 
HARDWARE CHECKLIST 


Central Processing Unit (CPU) 
1. Organization (word and/or byte oriented) 
2. Processor storage characteristics: 

Real, buffered or virtual processor storage; core or 
monolithic; amount reserved for firmware; net amount avail- 
able for operating system and problem programs. Amount of 
low-speed storage included, if any. 

3. Complement of registers 

4 Memory cycle time 

5. Average “access to processor storage" time 

6. Number of words or bytes accessed per cycle 

T. Instruction repertoire 

8. Instruction mix timing (average execution time) 

Example: (5-byte unpacked fields) 

8. 0 ۶ ۵ t D 

Db. C =a + Db 

em Cc = a tb 

d. Move a to b 

e. Compare a to b and branch 
instruction mix should be chosen based on expected use, For 
instance, if a significant amount of floating-point work is 
expected, then these instructions should also be timed. 

If the arithmetic instructions are performed in the 
registers, the loading to and storing from registers should 


be included in the timings. 
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9, Special power unlt required 
mo. 1/0 channels 
a. Number of channels by type (selector, multiplexor, 
or block-multiplexor) 
b. Maximum speed of each 
c. Attachable units (or excluded units) 
d. Switehing eapability of attachable units 
e. Simultaneity of operation between CPU and the 
I/O units, as well as between the I/O units themselves 
f. In-board channel (CPU acts as channel processor) 
or out-board channel (channel processor separate from CPU) 
g. Channel diagram of proposed system 
h. Attachable to another CPU 
i. Integrated controllers 
a. Attachable I/O units 
b. Limitations on which integrated controllers may 
or may not be core resident 
c. Degradation of CPU performance caused by the 
integrated controllers 
12. Timers/clocks 
a. Resolution or precision 
b. Maximum time accumulation 
Qc uterrupta dccem 
d" Ee ስመርርር 


e. Time of day or interval timers 





B; 


13. 


14 


15 


16 


e 


e 


Power failure protection 

a. Emergency off-automatic shutdown sequence 
e Comer 66 

2፡09. vandby or Secondarmipover source 
Storage protect capabilities 

a. Number of separate areas protected 
b. Fixed areas or software controllable 
c. Minimum area protectable 
Compatibility/emulation features 

a. Machines emulated 

9۰ Software requirements 

c, Limitations 

Expandability 

a ||: |. ۳ 6> 


b. Maximum storage and channels 


Magnetic Tape Units 


ECCO A‏ انا 


Number of units 

Number of controllers 

Densities supported, single or dual 
(-Irack/9-Irack 


Operating characteristics: Mounting operation (auto- 


load or manual), tape cartridge required or usable, fixed or 


rotatable dial, stress and wear on tape (number of capstans, 


vacuum column, tension arms) 


6. 


T 


S 


Continuous or incremental recording 
Transfer rate 


Skart /stop time 
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S. 


Rewind time 
Formula for computing effective speed 
Emwor=checkinge ana correcting capability 


Automatic or manual switching (between CPUs, channels, 


controllers) 


13. 


Expandability: maximum number of units per control- 


gem, controllers per CPU 


C. Card Read/Punch 


Ju 
B 
minute, 


D. 

1 

5 
2 
58 
sense, 
9. 
erint) 
TO. 


El, 


die 


Rated speed (reflects maximum speed) 

Time to process one card (converted to cards per 
this reflects minimum speed) 

Card codes supported 

Number of stackers and capacity of each 

Number of hoppers and capacity of each 
Error-checking capability 

Buffered, interlocking or cycle steal 


Special features: 51 column, punch-feed-read, mark 


and so on 


Capability for sorting, collating, interpreting (card 


Noise level 
Reliability 


Controller characteristics and limitations 


DE inter 


ከ 
2 
Sic 


Rated speed (for designated character set) 
Mae to print one line 


ymber of print positions 
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ti 


پک 
2 
E‏ 
,4 
.5 
I‏ 
channel‏ 
Uu‏ 
oe‏ 
El‏ 


Width of form (maximum and minimum) 

Quality of print (single and multiple form) 
Character set 

Skip speed 

Carriage tape specifications 

Lines per inch 

Noise level 

Stacker characteristics 

Reliability | 

Buffered, interlocking or cycle steal 


Controller characteristics and limitations 


Dusk or Drum 


Capacity 

Trans fer rate 

Access time (seek and rotational delay) 

Removable packs or fixed head storage 

Special features (such as rotational position sensing) 
Channel restrictions (such as attachable only to 
number one, or only device on the channel) 

Controller characteristics and limitations 
Expandability 


Rerjapı TWey 


F. Operator Console 


SE 


2 


6 6 ۶۰0۶۱ ۷۰ک 
Keyboard‏ 
Speed‏ 


۷۰۹٢٢٢ display 
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ES. 


Number of display lines visible to operator 
Character set supported 

Tocata vela iveco CRU and 1/0 units 
Noise level 

Reliability 

Special paper or stock form 

Stacker for paper 


Ribbon required-expected life 


G. Paper-Tape Reader/Punch 


e e 


o 


e 


O OO — GQ U 4. ላህ N BI 


بم 
© 


Speeds (transfer rate, start/stop time) 
T- or 9-channel tape 
BCD, EBCDIC or ASCII code 
Feed and take-up reels or fanfold 
Rewinding required 
necking capability 
X-on and X-off required 
Compatibility with source or destination of tape 
Splicing considerations 


Reliability 


H. Telecommunications 


ib, 


Controllers (data adapters) 

a. Number of lines supported 

b. Speed of transmission 

c. Leased line or dial-up 

ے١‎ ۰۰۰۰۰ 7607916105 775 
e. Types of terminals supported 


f. Interchange code supported 
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g, Features supported (such as paper tape, answer- 


back, auto-call, multiple-record transmission, pollirg) 


ne 11/01 

I ep len Oreniaer-cuptexeeranemission 
Ae Error correction/recovery 

Modems - See above and below 
Conmünication ٣٦ 

a. Leased or dial-up 

b. Multiplexed line 

c. Duplex or half-duplex transmission 
Terminal 

a. Tyre of display (CRT or hard copy) 
b. Input modes (such as keyboard or tape cassette) 
C. Speed 

d. ۶ 7 

e. Number of lines visible to operator 
f. Interchange code used 

E. special paper or stock form 

D. Impact or thermal printer 

i ENulviple copies 

j. Paper-stacking facility 

k. Intensity adjustment 
Ms E o ات‎ 

e‏ 66716 76-0 مر ٠>‏ ّ0 0 0 2۰6ھ 

n. Hard-wired or acoustic coupler 


«۰ On-line or- off-line transmission 
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۳ Other Equipment 

Many other types of equipment may be available to attach 
to or be used in conjunction with the computer System. Each 
requires various considerations regarding performance, suit- 
TITY for the purpose, compatibility with other units, 
reliability, operator interface and physical characteristics. 
Listed below are some of these types of equipment: 

im Microfilm/mieroficne 

2. Plotters/graphics 

3. OCR scanner 

U, Array processor 

5. Audio response 

er MICR 

7. Manual or automatic switching units 
Many other considerations such as power requirements, air 
بت 0د‎ 109٦11018, numlalrty comrol, floor space, and so on, apply 


to all the hardware. 
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APPENDIX C 
SOFTWARE CHECKLIST 


A. Operating System 
1. Resident device(s) 
2. Amount of direct-access storage dedicated to opera- 

ting system and work space required 
3. Processor storage reserved for operating system 
4. Support for anticipated I/O devices 
3 Extent of multioregramming capability and limitations 

6. Proposed method cf card I/O and print processing 
(SPOOL) 

T.  Preexecution I/O device setup 

8, Ease of operation 

8 707۹1016 ۴۹ ت9 ۰۰۹ 1360۹ 0060۱۶0۱0 5 دنس 

10. Access methods available 

11. Virtual storage-optional or required 

12. Support of automatic switching between channels 

13. Compatibility or emulation support-capabilities and 
limitations 

14. Complexity and capability of job-control cards/ 
language 

۱ Job=accounting facilities 

16. Operating system and hardware performance statistics 

17. Telecommunication facilities (Remote Job Entry, direct 


data entry/retrieval, time-sharing, and so on) 


18. Clarity of error codes/messages 
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Data-base management features 


12 1 dcs fon uper Program library 


B. Compilers/Assemblers 


0 
2 

3. 
D. 


D. 


Languages supported 

Adherence to national standard languages and features 
Processor storage required for execution 

Work space reguired on direct-access storage 


Maximum program size allowable (number of source 


statements) 


9 
D 
S. 
9. 
1. 
m. 


Devices not supported 

1/0 addresses absolute or generic 

Subroutine libraries available 

Suitability of languages to meet expected needs 
Telecommunication features 


Clarity of diagnostic codes/messages 


C. Sort/Merge 


T. 
2 
3 
4, 
> 
6 


T. 


Maximum/minimum file size 
Maximum/minimum record size 
Fixed/variable record lengths 

BISEKINE 

Number of fields in key-maximum key size 
Devices used/required/supported 


Formulas/tables to compute processor storage and I/O 


storage required 


۶۰۰۰٠۷٢٦٠٦٦٢٣7 ٣۶6 0 


ays 


2. 


List of utility programs available 


Completeness of list to meet needs 
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E. Performance 


L; 


er 
3 

43 
5. 


J. 


Estimate sort timings 

Estimate compile/assemble rate 

Estimate overating system overhead 

Estimate processing time of problem programs 
Estimate compatibility/emulation performance 


Predict toca wenen ከ O1 work load including 


operator functions and multiprogramming performances 


e 
ES 


Benchmark representative sample to confirm performance 


Use of simulation where advisable 


F, System Preparation Requirements 


1 


2 
B. 
7 
2 
6 
1 


SYSGEN plan 

On-site or remote 

Minimum system required to perform SYSGEN 
Amour.t of time required 

Degree of testing needed 

Vendor assistance 


Education required 


G. Software Availability/Reliability 


1. How long in use by other installation (or when 
available) 

2. Other users' experience 

3. Software maintenance 


a. Normal period between updates 
39-7 ۲10 س2 تا‎ 67۰60 ٣۱ 106 7 


c. Availability of vendor assistance 
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4. Quality and completeness of documentation 
5. Computer program patent considerations 
H. Vendor-Supplied Application Programs 
8۰۰۰ ር ዎስ ሚን تا‎ 
2. Programs required 


3. Programs not required but of potertial value 
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